Stack Exchange Network

Stack Exchange network consists of 175 Q&A communities including Stack Overflow, the largest, most trusted online community for developers to learn, share their knowledge, and build their careers.

Visit Stack Exchange

Questions tagged [googlebot]

Googlebot is the bot software that Google uses to crawl over 20 billion pages each day, the data obtained during a crawl is then analyzed and ranked on Google Search.

1
vote
2answers
27 views

How exactly does GoogleBot discover internal search result pages?

I know internal search result pages can be indexed sometimes but what I don't get is how it is crawled? Crawler only crawls links. Does that mean that someone would have to link to an internal search ...
1
vote
1answer
20 views

Does it matter if I use a different design elements for desktop and mobile versions of my site?

My question is about responsive designs (HTML + CSS). Does it matter if I use a different elements for desktop and mobile versions of my site? I mean that my page isn't really responsive, it only ...
1
vote
1answer
28 views

Do search engines index folders and subfolders regardless of pages?

I know search engines first index the main page, then follow by its links and do the same with the pages by the links as the main page before. Ok, but what about folders and subfolders? Do search ...
0
votes
0answers
19 views

Google Search Console reports crawl anomaly but the pages got traffic?

I am very confused to what is the current status of our website pages because the Google Search Console reports "Crawl Anomaly" and not-indexed status before and after using the Inspect URL tool - yet ...
1
vote
0answers
31 views

Google Search Console says: URL will be indexed only if certain conditions are met

I've got a few pages of my site that are coming up in G Webmaster Tools with errors. The pages are fully working, not restricted by noindex or robots.txt file. The pages don'r have a lot of content, ...
1
vote
0answers
20 views

How to improve Googlebot's indexing frequency? [closed]

I am experiencing low indexing issues on our website from several months. Google is not indexing the website fast enough. The changes we make take 1 or 2 months to get reflect in search engine. It's a ...
2
votes
1answer
40 views

Flag my website as a single page application?

Is it possible to flag my website as a SPA? The reason I want to do this is because on Google Search Console I see my links indexed without canonical links, but if I try for a live version the ...
0
votes
1answer
18 views

Is possible to prevent third-party links to web page from appearing in Google Search results?

We have a situation in which a sensitive website is blocked from being crawled using a robots.txt file. This works well, however the problem is that for a period of time the team used semantic urls in ...
1
vote
1answer
37 views

How to deal with IP redirection and Googlebot? [duplicate]

My client wants country-based domains for their customers. For example, all customers from UK will be taken to the .co.uk site, even if they visit the .com site. And same for other countries. I have ...
1
vote
1answer
28 views

Do I need to disallow mediapartners-google in robots.txt for SEO?

I found that most of the websites disallow mediapartners-google in robots.txt. Why do I need to do block mediapartners-google? Can blocking it help withp better SEO or does it not matter for SEO?
3
votes
2answers
78 views

AngularJS based SPA page indexing

I recently worked on a AngularJS based portal. One of the main areas of portal is to list available job opening for the organization. Right now, Google does index some of our job listings. Since jobs ...
2
votes
1answer
26 views

Do Google crawlers/extractors read attributes in React props?

We have a use case where we are displaying a certain product price when a page is rendered. And when a radio button is clicked on the page (product upgrades), we display additional prices. The "...
1
vote
1answer
39 views

Shrinking site's size by removing pages from sitemap and from internal links

We are an eCommerce site with about 50,000 pages. We decided to focus on about 2,000 so google will crawl our site better. We planning to remove 48,000 pages from the sitemaps and from the site links, ...
1
vote
0answers
21 views

How to verify that Googlebot can access the URL of a video thumbnail?

I need to check if Googlebot can see my image thumbnails. When I use the URL Inspection Tool, Google cannot see my images in the page. In the rendered screen shot, it displays a black box instead of ...
1
vote
0answers
53 views

Googlebot intermittently fails to download ressources from AWS CloudFront CDN

I have a weird issue and after a few weeks of investigation I think I may have found the cause : Cloudfront & Googlebot. Context : I launched a French website that compares and combines all ...
2
votes
3answers
290 views

Will google still index a page if I use a $_SESSION variable?

For a couple of pages on our site, I'm writing a widget that relies on what it displays on other pages to determine what it displays on the current page. Basically, the purpose is just to ensure there'...
0
votes
0answers
11 views

How to get whitelist my IP address in GMAIL? [duplicate]

How to get white-list my IP address in GMAIL?
1
vote
1answer
23 views

Google desktop crawler showing unreachable - mobile crawler is working

I noticed a sharp drop in Adsense earnings on one of my sites this week and after a bit of digging in the Search Console found errors suggesting there was a problem with my robots.txt. I checked this ...
3
votes
2answers
38 views

How to force a view of meta data as Google or Bing sees it?

I'm having an extremely peculiar problem with a site (WordPress) that just suddenly started exhibiting an issue where Japanese is showing up for all the metadata in Google and Bing - only: Google: ...
0
votes
0answers
31 views

Google is marking as duplicate my unique URLs powered by Angular Universal

I have a problem in the search console, and google is marking my unique urls as duplicate The application is build with Angular Universal and is rendered in the server Note: This site is only ...
0
votes
0answers
34 views

SEO Ajax loaded pages - noindex X-Robots-tag

I have on my homepage (page A) html content (ajax) loaded on click of a button (load more) from multiple pages (page B). Googlebot does see content and does follow links loaded from the ajax calls on ...
0
votes
0answers
23 views

Googlebot crawl anomaly due to blocked resources by Zopim

I am using Zopim live chat to enable chat support in my website, but Googlebot is showing error due to some unreachable resources from zopim service. I have attached a screen capture of the error. ...
5
votes
1answer
147 views

Cookie consent overlay - Does it affect SEO?

I have a german website with Adsense integrated. To meet Google's compliance requirements, I ask users to accept cookies for personal advertising. To achieve this, I created an overlay which stays in ...
2
votes
1answer
43 views

Is blocking googlebot from accessing portions of the site with robots.txt penalized by Google?

I was reading this question. Let's say i have a div with something useful for users but i don't want it to be considered by Google. If i put this on an iframe and block the googlebot with robots.txt ...
1
vote
1answer
47 views

Is crawl budget and link juice being wasted to URLs with hashes?

On my site I have internal links to URLs with hashes in them, like this one: https://example.com/page.html#music https://example.com/page.html#music has canonical tag with URL pointing to https://...
0
votes
0answers
27 views

Googlebot can't index my hbs based SPA website

As part of a school project, I have made a single-page application website using server-side rendered handlebars. However, when inspecting the site using Google Search Console, all Google can find is ...
0
votes
0answers
26 views

Referencement issue with irrelevant suffix

I have a problem concerning the "coverage" on Google Search Console. The website on the company I'm working for have weird errors on this page. Google is telling that it had issues referencing pages ...
3
votes
1answer
37 views

Would using mouse movement to change how the page is rendered be considered “cloaking” to Google?

My site can enhance itself to client side rendered (CSR) (ish) if a prediction can be made by mouse movement analysis. It can gauge a visitor's familiarity with the site, confusion, etc. The content ...
1
vote
1answer
24 views

Will keywords written into a page using JavaScript replacing placeholders in the page source affect SEO?

Website pages use SEO content (words) that are dynamically generated. Like the store name (on a multi store site). However the dynamically generated words only show when you inspect element, as below....
1
vote
1answer
140 views

Google's Inspect URL shows incorrect source code for my page's “Google Index”, but correct for “Live Test”

My web page is incorrectly indexed and is showing a different header than the one that is actually updated in my website. When I used the URL Inspection Tool in Google Search Console, The index shows ...
1
vote
1answer
131 views

Crawled page's screenshot is blank in the Google Search Console

I set up a SPA page through Angular, and achieved the effect of Server Side Render through Rendertron, In the facebook debug tool, I could get my dynamically generated Meta data, but the preview of ...
2
votes
1answer
63 views

Redirecting a user but not Googlebot to solve problem with iFrames in content

TLDR To work around a problem with a third party documentation writing software I need to redirect a page to another page but only if a user in browser is looking at it. I want the google bot to see ...
3
votes
1answer
2k views

New Google Search Console does not load any resources such as CSS and JS while it worked fine in the old Search Console tool

When using URL inspection and Live Test in the new Google Search Console the Googlebot does not load most resources (Javascript, CSS and Images). This results in the Search Console telling me my pages ...
1
vote
0answers
24 views

When using an image server on a separate domain, should we link directly to the image URL and restrict crawling, or redirect through our domain?

I have a large number of images stored on a separate server, these URLs currently go through a 302 redirect. I think these additional pages are a total waste of time for Google to crawl. At the ...
2
votes
2answers
44 views

Do non-existent 'category' pages in URLs present a problem for search engines?

Say for example if I have a news site and my URL structure is: example.com/news/random-article If there is no page at example.com/news/, in other words it results in a 404 error, is that problematic ...
3
votes
1answer
53 views

Google not showing cache option on the following url in SERP results?

Google is not showing option to check cache on the following url that I have mentioned below in SERP results.
2
votes
1answer
29 views

Can I use robots.txt to allow Google to access some folders, but disallow everything else?

I want to allow Google bots to crawl only my index.php and the content of some folders. Would that work? User-agent: * Allow: /index*.php Allow: /folder1/ Allow: /folder2/ Allow: /folder3/ Allow: /...
1
vote
1answer
124 views

Why are some google snippet results bold? [closed]

Recently, when I search for keyword "best mlm software" I got a featured snippet having "List of Multi Level Marketing Software". I noted that two of list's item were BOLD text. Didn't know why??? Can ...
1
vote
1answer
52 views

Does Google Crawl Rate have an impact on Google's Image Bot

I have a relatively large website. Each page has several images. Google indexes my website very slowly, but it actively crawls my pictures. Does changing the Google Crawl Rate have an impact on Google'...
2
votes
1answer
72 views

Is it good for SEO to change an article's slug when changing its title?

I have the following URL view: example.com/article/{slug}-{id} If I change the title of the article, then the slug will also change so that there is no disagreement. Please note that the article ...
2
votes
2answers
37 views

How many levels of sub-pages do Google bots crawl to when fetching and indexing linked pages in Google Search Console?

Let's say I submit a sitemap page that contains all the links I want Google to crawl, but instead of submitting a sitemap, I fetch the page and linked pages and let Google index them. How many levels ...
2
votes
0answers
305 views

Discus comments cause “clickable elements too close together” in Google Search Console mobile usabality report

I just got an email from Google that they found some problems on my site: clickable elements too close together Content wider than screen I fetched problematic URLs as Googlebot and noticed that it ...
1
vote
1answer
45 views

Can Google crawl .TS (MPEG2-TS) video format?

We are using the tool mp4hls (https://www.bento4.com/developers/hls/) to prep the hls stream. Basically, I think it can conver any video format to .mp4 then it convert to .ts. However, Google only ...
1
vote
1answer
39 views

How to get Google to crawl our product pages as soon as they are updated for in-stock or out-of-stock?

We have an automobile website where we have different cars and bike models. We have product pages for every car/bike. Each page has the products price, Offers, reviews , news, images and videos. ...
1
vote
1answer
56 views

Google Bot Crawls using expired cookies and fails to get response

We are using a cookie on our website which is to fetch some information through an API. The cookie becomes invalid after a very small duration. Google Bot uses the same cookie again and again to ...
1
vote
0answers
32 views

Does Google index ajax data that load after 1 second by setTimeout?

I use load Javascrpt and use Ajax for load content. And i should use setTimeout for run Ajax load function after 1 or 2 second(s). Does Google index and follow this content?
1
vote
0answers
14 views

Download link expired, going back to software presentation page, what redirect method to use?

When someone (user or bot) reaches an old download link like /download/yellow-wallpaper.zip?key=some-expired-download-key , user is sent back to file details page, e.g /yellow-wallpaper.html, where ...
1
vote
1answer
22 views

SEO - Custom error pages & status codes

My website uses a custom error page with the <meta name="robots" content="noindex"> meta tag. Am I right in thinking this tag will tell crawlers, such as Googlebot, to remove the page from ...
1
vote
1answer
81 views

Organic search traffic - big discrepancy between users and new users

Background: We are a non profit site only really acquiring traffic through paid ads. Our SEO is only just getting started so we don't get a high level of organic traffic. Recently we've seen an ...
1
vote
1answer
26 views

Alternative languages redirecting

Serving a page in English, EG: https://www.example.com/en/make-games Has the following meta tags: <meta rel="alternate" href="https://www.example.com/fr/make-games" hreflang="fr" /> <meta ...