We’re rewarding the question askers & reputations are being recalculated! Read more.

Questions tagged [googlebot]

Googlebot is the bot software that Google uses to crawl over 20 billion pages each day, the data obtained during a crawl is then analyzed and ranked on Google Search.

Filter by
Sorted by
Tagged with
1
vote
2answers
2k views

Googlebot flooding server with requests for junk URLs with random data

I'm having some trouble with GoogleBot. It keeps requesting a random URL that doesn't exist. It is trying to access:www.example.com/index.php/{TOKEN} That {TOKEN} is really random, no idea where it ...
1
vote
1answer
58 views

Website disappeared from Google after redirecting Googlebot to HTML with only the metadata I want indexed

I have a website on which I use a redirect only for Googlebot to a HTML file. It only contains metadata that I want to show in the search results. I have removed everything else from the HTML: <!...
2
votes
1answer
427 views

Can I disallow JS scripts in robots.txt for Googlebot if the website degrades to offer equivelant non-JS functionality?

To my knowledge, Googlebot is currently fully capable of rendering complex SPA applications, and this is recommended as a rule of thumb. The website is currently routed from server side and served as ...
3
votes
1answer
333 views

Does Googlebot execute Google Tag Manager?

I wanted to understand how Googlebot (and other crawlers) crawls my site. Specifically whether it passes a document.referrer and if it maintains localStorage keys, so I implemented a script via Google ...
2
votes
2answers
338 views

AMP pages not getting updated when served from Google's cache

I have an amp page at the following UNS and its already cached by Google. When this paged is served from Google's cache, its showing a 2 day old version of page. Original AMP URL http://www....
2
votes
1answer
104 views

Prevent “Fetch as Google” from creating a wishlist in the database of a Woocommerce site

Can anybody explain why when I 'Fetch as Google' and ask it to index a new page, Google creates a new Wishlist on the back end of the site? Is this normal? The site was built by Yell and we have had ...
1
vote
1answer
155 views

404 or 302 Redirect - what to use for a url which may be used in the future but not available at the moment

My site lists blogs like this example.com/?status=blog&id=number I only have 3 blogs now (id=1, id=2 and id=3). I checked my log files and saw that search bots are manipulating the URLs and ...
3
votes
1answer
273 views

Google doesn't recognize a 404 status code

There are 404 pages with two kinds of response headers (copypasted in full length from Chrome DevTools, Network tab): Response headers: cache-control:max-age=0, no-store content-type:text/html date:...
1
vote
1answer
359 views

Website Home Page URL Not Show On Google Search [duplicate]

If i search like site:www.paka.tv my main homepage URL not showing, but categories of my site pages showing on results. How to show my website URL (www.pakka.tv) on google search.
5
votes
0answers
1k views

Search Console: Server Connectivity Error - Connect Timeout

Over the last 4 days server connectivity errors have started to pop up in my Search Console. Specifically, it is the connection timeout error. I have been talking to my hosting provider and they ...
1
vote
1answer
1k views

How to find out the referrer of Googlebot's crawling URL?

Googlebot crawls 100s of 404 URLs from my website. I want to know from where it gets those links? Is there anything like HTTP Referrer? In detail: Googlebot crawls example.com/no_such_files.php ...
0
votes
0answers
24 views

How to handle the duplicate block of content?

Please refer: https://support.google.com/webmasters/answer/66359?hl=en Googlebot don't like duplicate content and also duplicate block of content repeatedly copy and pasted in many pages. I think, I ...
2
votes
1answer
42 views

Can we have Google crawl but ignore our paginated category pages and prefer our individual post pages in the index?

We have a site with a massive content back-end (50.000+) with, lets say, posts. We want to make these posts available to users, so people searching for the content on Google will find one of these ...
1
vote
0answers
448 views

If “_escaped_fragment_” shows up in Search Console URL Parameters, should you “Let Googlebot decide”?

As far as we know, there should be no links to our site anywhere that include the _escaped_fragment_ parameter used for fetching HTML snapshot versions of our JavaScript-heavy pages. We use the ...
0
votes
1answer
144 views

How frequently should I expect Google AdsBot to visit my site?

I bought paid advertising for my website through an internet advertising company. I noticed that, among all the visits coming from that source, a large amount of them have AdsBot-Google (+http://www....
2
votes
1answer
573 views

htaccess empty referer deny “google bot”

I put this rule into htaccess file to deny empty referers which is returning 403 SetEnvIfNoCase Referer "^$" bad_user Deny from env=bad_user I can see the log, it's denying also Googlebot which ...
1
vote
1answer
444 views

Google crawler 520 error

Google Crawler generates 520 errors daily. I will get about 50-100 of these errors in a 60.000 internal links website. The thing is though, that my links don't have any issues that I can dig into ...
1
vote
2answers
473 views

Should an Allow or Disallow directive be used in robots.txt to allow Googlebot to crawl the whole site?

User-agent: * Disallow: / User-agent: Googlebot Allow: / I use this command in my robots.txt file. But i do not think this is right. What should be the right command? Some article i have found where ...
1
vote
0answers
453 views

How to stop Bot generate Traffic in Wordpress?

My WordPress website recent facing a big problem. Spamming problem. Using a plugin i detect that every single second here coming huge number of traffic which is fake and send by BOT. The traffic also ...
1
vote
2answers
171 views

Does Googlebot add any parameters to a URL when crawling?

I have set up a memory caching system that will use the URL as a key to cache a copy of a HTML page in memory in order to make it faster. The result is bringing page load down from around 4 seconds to ...
1
vote
1answer
67 views

Shutdown a subdomain can affect another subdomain

I have two subdomains and I decided to shutdown the br and focus only in pt project. https://pt.example.com https://br.example.com My question is that I have some old posts (archived and not ...
1
vote
1answer
602 views

How do I get Google to crawl and index a react.js app that uses AJAX and onclick events instead of links?

What would be the best way to get Google to index a react app. Google can now read JavaScript, so no pre-rendering or isomorphic application is needed. But what about internal links on a page? We use ...
1
vote
3answers
549 views

does google crawl/follow link on a page with meta robots noindex

We have added following meta tag on some of our pages: <meta content="noindex" name="robots"/> Does google crawl the content and links on this page and just don't show it on the google search ...
2
votes
1answer
95 views

Re-index updated post and update cached AMP version

I have a post I just updated on a site. This site gets picked up in google almost instantly but this post that I updated is not getting updated. As a matter of fact the post in mobile index, which is ...
3
votes
2answers
66 views

Versioned pages: making sure Google points to the current [duplicate]

The open-source Sinon project has some docs that are versioned per-release. That means that when searching for sinon documentation, you might end up at a doc for version 2.4.1, instead of the latest ...
2
votes
2answers
382 views

What is the way to prevent Google from scraping and crawling a URL in plain text on a page?

Problem I have a website with documentation that includes examples with plain text URLs. The other day, I noticed that Google Webmaster was telling me that one of those URLs generated a Page Not ...
1
vote
1answer
104 views

Remove cached domain / pages from Google

The website's domain I work for is https://www.utazzitthon.hu and there is an other company in partnership with us, they have a domain that is directing to our server and our content, but with this ...
1
vote
1answer
1k views

Googlebot and Mediapartners-Google are crawling URLs blocked by robots.txt, why?

I have blocked the some of the URLs from robots.txt. I think there is No way to access the blocked URLs from my website. But I have seen in my logs, whatever I blocked the URLs from the robots.txt are ...
1
vote
0answers
466 views

Does Google crawl AJAX API calls?

We have been monitoring our logs for Googlebot and there are many requests to API calls (requests via AJAX) which are used to fetch the page and not the complete page url. How does Google ...
1
vote
1answer
88 views

For SEO, are pre-rendered previous and next pages required for a single page application site with infinite scroll?

In this article It is mentioned that Chunk your infinite-scroll page content into component pages that can be accessed when JavaScript is disabled. and Add rel="next" and rel="prev" in head ...
1
vote
2answers
59 views

Are rotating ads harmful for Google SEO?

I have a plugin which makes different banner ads to be displayed randomly. It means that every time a page is reloaded, a different code is also reloaded. To make it more complicated: One banner ad ...
4
votes
2answers
165 views

How does Google bot know that my page content has changed?

Let's say I've a page which is already indexed in Google Search. If I make a change to the content of that page, how would Google bot notice and updated it in the Search? Does he compare the indexed ...
2
votes
0answers
239 views

Google cache does not have prerendered _escaped_fragment version of page, rather it is showing another page altogether

I have read multiple questions around it but still could not find a solution. We have a page in the form https://example.com/trending. We are using Prerender using the ?_escaped_fragment url to show ...
-3
votes
2answers
69 views

How to block some CSS files from Googlebot using robots.txt while allowing other files in the same directory?

Hi in my Drupal website site I have a few directories like: /sites/modules/ Now I only want googlebot or other bots to index or allow say file user.css I don't want Googlebot to index the file ...
1
vote
1answer
320 views

Prevent Googlebot from crawling “access denied” error (403) of private forum that are reported in Google Search Console?

I'm running a website base it by vbulletin. Recently I moved one of my forums to a private section of my website so users and googlebot cannot access it there. Now I'm getting lots of access denied ...
2
votes
1answer
474 views

The Googlebot is only fetching my website's assets [duplicate]

When I try to use Fetch as Google on pages for my website, the Googlebot fetches it partially, and when I check the crawl log, tells me that some of my assets are temporarily unreachable: What should ...
1
vote
1answer
280 views

Prevent indexing of site search results

Users have the ability to search on my site. This function renders a search results page that has occasionally been indexed by google and served in SERPs. Normally this is not an issue, but when ...
1
vote
1answer
470 views

Internal Search Results: NOINDEX or robots.txt Blocking?

We have our own internal search results pages for our download section. These results are currently crawlable but with NOINDEX meta tags so they don't appear in the search index. People link to ...
2
votes
1answer
97 views

How to get my # links on my single page site indexed by Google

I own a One Page Site. Google is only indexing the home page -> example.com. I tried adding it in Google Webmaster Tools/Search Console how it should be displayed: http://example.com/#ueber-desjash-...
1
vote
2answers
94 views

Would an online queueing system for a website have an effect on google's ability to crawl the site?

If I was to implement a visitor queuing system for my website, where if visitor count is higher than n, place them in a queue. Would this have an effect on google's ability to crawl the site? Thanks
1
vote
2answers
633 views

How to Crawl a website requires cookies for audit?

Situation: My Client's website requires cookies to access it. Users should choose (Language and country) to access the website. The problem is: Whenever I try to crawl the website using any software (...
1
vote
1answer
177 views

GoogleBot is scanning random link on my website

Just take a look at this: my-domain.com:80 66.249.79.18 - - [13/Jun/2017:20:09:26 -0700] "GET /d733421/my-domain.com HTTP/1.1" 404 16729 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www....
1
vote
1answer
154 views

Disallow, canonical, or noindex for a coupon ID parameter where Google has already found some duplicate content?

We've changed our software and domain a week ago. Done the 301 redirction successfully. However, we're still facing a lot of problems, bugs, things we didn't think about and more... Our website is a ...
1
vote
1answer
40 views

Linked desktop and mobile pages- does SEO rank of mobile website even matter?

The question is somewhat similar to this post. The only difference is, we have desktop version as https://www.example.com/ and mobile version as https://www.example.com/m/. The desktop page has mobile ...
1
vote
1answer
52 views

Link to product detail pages without allowing product list pages with many filter variants to all be indexed

I have a website for managing trading card game (TCG) collections which is built around a central search engine/filter system. This filter system is located on the cards page which lists all cards ...
5
votes
1answer
853 views

Google crawler and dynamic websocket website

I have a a big problem that I'm only considering after I have finished development of a webpage. My page is written with only a skeleton HTML. All the actual content of my site is sent through ...
2
votes
1answer
783 views

Why is Google crawling non-existant URLs?

I can see in live traffic of my wordpress website that goggle bot crawl non existing pages. www.example.gr/search/search-results/password-reset%252Fpassword-reset/password-reset%252Fpassword-reset%...
0
votes
1answer
91 views

How can I prevent the Googlebot from indexing duplicate URLs?

I have duplicate URLs for each ad page. For example: https://example.com/ads/2161/%D9%84%D9%83%D8%B2%D8%B3_GS350.html https://example.com/2161/ I used the second URL for Twitter and Instagram posts....
0
votes
1answer
561 views

Can i do a 301 redirect for the Googlebot Only?

Lets say a have a webpage called example.com, and a have a subdomain called sellingpage.example.com I use sellingpage.example.com as a landing page for all my paid traffic, this is a landing page ...
1
vote
1answer
2k views

Firebase hosting, sitemaps and robots.txt

i tried searching on internet about this question and how Firebase handle indexing and how owner of the content running on Firebase hosting can influence crawling by Google bot. Personally, i am ...