Questions tagged [googlebot]

Googlebot is the bot software that Google uses to crawl over 20 billion pages each day, the data obtained during a crawl is then analyzed and ranked on Google Search.

Filter by
Sorted by
Tagged with
2
votes
2answers
21 views

How does google treat h1-3's that are styled to be smaller than a lower class of heading?

For example: You have you h1 <h1>Epic SEO Keywords</h1> then right below <h2 class="Massive text">Enjoy the future of you, visualise your dreams</h2> Will Google render ...
3
votes
1answer
42 views

How to change Google crawl rate in new Google search console?

Google has now completely removed the old search console (https://searchengineland.com/the-old-google-search-console-is-no-longer-available-321650) where you used to be able to request a custom crawl ...
3
votes
2answers
74 views

Can Google crawl dynamic structured data added in a window.onload() event?

I inject some JSON-LD into a page dynamically via Javascript, when I test the page using Google's Structured Data Testing Tool, the expected output appears and I can see the Product Element is ...
2
votes
0answers
20 views

URL crawlable in old Google search console, but blocked in new

Google stopped crawling images for our site, when we moved images to cdn subdomain. Old search console states image is crawlable, but new one states it is blocked by robots.txt (for same link). our ...
3
votes
1answer
45 views

Can I hide ads for Googlebot so it loads my site faster?

I have ads which load sometimes slower, I want to hide those ads for Googlebot. I plan to hide that ad code for googlebot only and show them only to users. Do I risk a Google penalty if I hide ads, ...
1
vote
0answers
34 views

Crawl anomaly caused by googlebot crawling secure_link mp4 files with md5 and expires parameters

Googlebot continues to crawl mp4 files in the format www.example.com/video/12345.mp4?md5=2029zanlkdain312on&expires=1523327783 which has obviously expired (only valid timestamp+30minutes, after ...
3
votes
2answers
76 views

Googlebot keeps on crawling stale/nonexistent(410) resources and then shows a crawl anomaly followed by de-indexing pages

We have been experiencing constant crawl anomalies caused by googlebot continued crawling of 'rnd.js?asdfasdfasfs3423' (hash is random on each pageload). It has been 3 months since we removed rnd.js ...
3
votes
0answers
33 views

Pantheon Dev Site Trapped in Google Index (with injected robots.txt)

We have a bizarre scenario I'd like opinions on. A site at example.com is being built on pantheon, using their example.pantheon.io staging / dev URL*. Pantheon's development platform injects a ...
1
vote
1answer
59 views

Google is ignoring meta tags and robots.txt

I have the following meta tags on my website (dev server): <meta name="robots" content="noindex"> <meta name="googlebot" content="noindex"> And I also configured a robots.txt (...
0
votes
0answers
9 views

Google Search Console says URLs blocked by robots.txt even after robots.txt is updated [duplicate]

Yoast SEO had created a sitemap and robots.txt that did not suit my page. I have since replaced both and waited a few weeks but my robots.txt is still blocking 13 pages. Can anyone help me with what I ...
2
votes
3answers
55 views

Stop googlebot crawling URL more than once?

I have a site that usually creates a few thousand pages a day, which don't change after they have been created. Recently my dedicated server has crashed due to googlebot crawling the site too often. ...
0
votes
0answers
9 views

How to submit and get links indexed faster on google search console? [duplicate]

I've created a website which has a lot of links. I've added it to google search console in order to show them on google search but it is indexing around 2,000 links every week. I have 200,000+ links. ...
0
votes
1answer
46 views

Robots.txt, php.ini, .htaccess, etc

This will be my first website, so I am completely new to all of this. I cannot seem to find an answer anywhere, as to whether or not I should disallow configuration files like /php.ini or hidden ...
0
votes
0answers
5 views

What are the seemingly obvious things to improve search engine rank? [duplicate]

One thing that seems like the most obvious thing I can think of, at least when it comes to businesses, is to register your business with Google My Business. I can't think of anything more obvious that ...
1
vote
1answer
50 views

How to resolve Google “Indexed, though blocked by robots.txt” warning for a user login protected page?

Or, in other words, how can I tell google this is a user login protected page? The website is basically a single page app with server side rendering. That said, if the visitor is asking for a web ...
2
votes
2answers
29 views

Will I get penalized by Google for cloaking if I redirect mobile users to a sub domain?

I have a website https://example.com which is responsive but it came a time where it displays too much text. So I decided to create a progressive web application version and would like to deploy it ...
1
vote
1answer
37 views

Which page speed google SEO uses? [duplicate]

My SEO team told me Google SEO engine uses the page speed in its algo to rank the pages. It is confirmed by this blog too. In my case, page speed is different for end-user coming from browser than ...
1
vote
0answers
31 views

Huge differences between webmaster tools' crawl stats and apache logs

We noticed huge difference in the metrics. For example, last saturday : According to Webmaster Tools : 6,082 pages crawled 76,119 kilobytes downloaded 1,550 ms in average to download a page Now ...
0
votes
0answers
44 views

Angular 7 - will removing hash on my angular app make my website SEO friendly?

I have an angular website thats been running for more than 2 years but sadly when i checked, only the homepage was indexed. My about us, contact us page which are serving static content was not also ...
3
votes
1answer
61 views

GoogleBot crawls thousands of urls like 2487763877595434670.htm that doesn´t exist

in our serverlogs there are thousands of requests (50-100k) from googlebot every day for urls like /2487763877595434670.htm (as I can see always 19 random digits with .htm at the end) First the bot ...
1
vote
1answer
18 views

How to stop google from indexing old URLs of a website which redirects to new URLs

My old WordPress configuration added index.php to the permalinks of all my pages. e.g: https://example.com/index.php/contact etc., It was indexed already by google crawler. Now I reconfigured ...
0
votes
0answers
10 views

Get Google to forget site [duplicate]

Is it possible to make Google "forget" a site and reindex it? I have a site that I changed dramatically, and I want to do SEO on my new site, but Google isn't deindexing the old site. How could I do ...
2
votes
1answer
44 views

GoogleBot doing POST requests

Why has GoogleBot started last Friday to do POST-requests on a page. I can see in the logfile (just an example, had about 10.000 entries over the weekend - the url in the log is changed): 66.249.79....
0
votes
0answers
20 views

Should I be worried about 461K pages crawled but currently not indexed [duplicate]

I found a huge number of crawled and not indexed pages in my Google webmaster panel. On the other hand, I had 220K valid links. When I searched about most of articles which were listed in the ...
2
votes
1answer
29 views

Google Custom Search showing redirect URLs to external links despite rel=“nofollow noopener” on those links

I have a website which contains a list of all businesses in my region. They all have a profile page that can include a Facebook or website URL. I added rel="nofollow noopener" to all external links ...
1
vote
1answer
16 views

Can i use 301 redirect for sitemap.xml file location

I want to keep the autogenerated sitemap.xml on a different server. Will google follow 301 redirect while trying to fetch sitemap.xml? https://example.com/sitemap.xml --301--> https://static....
0
votes
0answers
31 views

Google Search Console - Verification Failed

I am trying to add my website to Google Search Console but failed, it returns The connection to your server timed out The file is there, I can open it on the normal browser, all meta tag set to ...
0
votes
0answers
18 views

Googleboot keep crawling non-existing page, not my actual page

I started to build my website 6 months ago. Since I started until now for 6 months. Googlebot keep crawling non-existing page on my site. Googlebot randomly generates those links. They accessed to ...
0
votes
0answers
13 views

How does googlebot indexing javascript mixed links (href & onclick on same element)

It's been about 5 years since google indexing javascript and I'm still confused about details. I read here ( Does Google perform onClick actions? ) basics. And then I know that javascript created ...
0
votes
0answers
86 views

Serving static HTML pages generated using Rendertron - Is it considered cloaking?

Preamble Hello, I'm trying to improve SEO of a web app built with Angular 7. Unfortunately, I can't (not allowed to) install node on the server it's being hosted on, so I cannot implement server-...
1
vote
2answers
38 views

How exactly does GoogleBot discover internal search result pages?

I know internal search result pages can be indexed sometimes but what I don't get is how it is crawled? Crawler only crawls links. Does that mean that someone would have to link to an internal search ...
1
vote
1answer
29 views

Does it matter if I use a different design elements for desktop and mobile versions of my site?

My question is about responsive designs (HTML + CSS). Does it matter if I use a different elements for desktop and mobile versions of my site? I mean that my page isn't really responsive, it only ...
1
vote
1answer
33 views

Do search engines index folders and subfolders regardless of pages?

I know search engines first index the main page, then follow by its links and do the same with the pages by the links as the main page before. Ok, but what about folders and subfolders? Do search ...
0
votes
0answers
60 views

Google Search Console reports crawl anomaly but the pages got traffic?

I am very confused to what is the current status of our website pages because the Google Search Console reports "Crawl Anomaly" and not-indexed status before and after using the Inspect URL tool - yet ...
2
votes
0answers
713 views

URL will be indexed only if certain conditions are met

I've got a few pages of my site that are coming up in G Webmaster Tools with errors. The pages are fully working, not restricted by noindex or robots.txt file. The pages don'r have a lot of content, ...
1
vote
0answers
26 views

How to improve Googlebot's indexing frequency? [closed]

I am experiencing low indexing issues on our website from several months. Google is not indexing the website fast enough. The changes we make take 1 or 2 months to get reflect in search engine. It's a ...
2
votes
1answer
67 views

Flag my website as a single page application?

Is it possible to flag my website as a SPA? The reason I want to do this is because on Google Search Console I see my links indexed without canonical links, but if I try for a live version the ...
0
votes
1answer
22 views

Is possible to prevent third-party links to web page from appearing in Google Search results?

We have a situation in which a sensitive website is blocked from being crawled using a robots.txt file. This works well, however the problem is that for a period of time the team used semantic urls in ...
1
vote
1answer
52 views

How to deal with IP redirection and Googlebot? [duplicate]

My client wants country-based domains for their customers. For example, all customers from UK will be taken to the .co.uk site, even if they visit the .com site. And same for other countries. I have ...
1
vote
1answer
78 views

Do I need to disallow mediapartners-google in robots.txt for SEO?

I found that most of the websites disallow mediapartners-google in robots.txt. Why do I need to do block mediapartners-google? Can blocking it help withp better SEO or does it not matter for SEO?
3
votes
2answers
117 views

AngularJS based SPA page indexing

I recently worked on a AngularJS based portal. One of the main areas of portal is to list available job opening for the organization. Right now, Google does index some of our job listings. Since jobs ...
2
votes
1answer
43 views

Do Google crawlers/extractors read attributes in React props?

We have a use case where we are displaying a certain product price when a page is rendered. And when a radio button is clicked on the page (product upgrades), we display additional prices. The "...
1
vote
1answer
47 views

Shrinking site's size by removing pages from sitemap and from internal links

We are an eCommerce site with about 50,000 pages. We decided to focus on about 2,000 so google will crawl our site better. We planning to remove 48,000 pages from the sitemaps and from the site links, ...
1
vote
0answers
26 views

How to verify that Googlebot can access the URL of a video thumbnail?

I need to check if Googlebot can see my image thumbnails. When I use the URL Inspection Tool, Google cannot see my images in the page. In the rendered screen shot, it displays a black box instead of ...
1
vote
0answers
81 views

Googlebot intermittently fails to download ressources from AWS CloudFront CDN

I have a weird issue and after a few weeks of investigation I think I may have found the cause : Cloudfront & Googlebot. Context : I launched a French website that compares and combines all ...
2
votes
3answers
334 views

Will google still index a page if I use a $_SESSION variable?

For a couple of pages on our site, I'm writing a widget that relies on what it displays on other pages to determine what it displays on the current page. Basically, the purpose is just to ensure there'...
0
votes
0answers
22 views

How to get whitelist my IP address in GMAIL? [duplicate]

How to get white-list my IP address in GMAIL?
1
vote
1answer
28 views

Google desktop crawler showing unreachable - mobile crawler is working

I noticed a sharp drop in Adsense earnings on one of my sites this week and after a bit of digging in the Search Console found errors suggesting there was a problem with my robots.txt. I checked this ...
3
votes
2answers
41 views

How to force a view of meta data as Google or Bing sees it?

I'm having an extremely peculiar problem with a site (WordPress) that just suddenly started exhibiting an issue where Japanese is showing up for all the metadata in Google and Bing - only: Google: ...
0
votes
0answers
39 views

Google is marking as duplicate my unique URLs powered by Angular Universal

I have a problem in the search console, and google is marking my unique urls as duplicate The application is build with Angular Universal and is rendered in the server Note: This site is only ...