Stack Exchange Network

Stack Exchange network consists of 174 Q&A communities including Stack Overflow, the largest, most trusted online community for developers to learn, share their knowledge, and build their careers.

Visit Stack Exchange

Googlebot is the bot software that Google uses to crawl over 20 billion pages each day, the data obtained during a crawl is then analyzed and ranked on Google Search.

1
vote
2answers
22 views

Ampersand (&) in actual URL and sitemap

I have a sitemap (which is submitted to Google for indexing) in which I am adding URLs which have an ampersand (&) in it. Since in sitemap & has to be escaped I replaced & with & ...
0
votes
1answer
7 views

Redirected page need to make no-index on google

All images used in WordPress website have got a URL and WordPress automatically created a URL for a landing page using the image.php file. So protecting to index all images with dedicated pages URL I ...
0
votes
2answers
25 views

Is Google Bot indexing the content of a textarea?

For one of my projects that provides pages with different content in a textarea, I need to make sure that Google Bot is indexing the content of those textareas. I could not find an answer online. ...
1
vote
1answer
23 views

Google started to reports root url blocked by robots.txt - but it's not -

My website has been running for multiple years without problem, however I recently noticed my root url on Google does not show any description/title. The Webmaster Tool reports this error: Crawl ...
1
vote
0answers
10 views

Why the Download Pages of our products are not indexed by Google? [duplicate]

I have a question about pages indexed, hope you can help me out, thank you in advance. Recently, we found our French site's two download pages are not indexed by Google. The URL of the French site ...
1
vote
0answers
41 views

Security Issue - Harmful content

I am trying to solve an issue with my website already for some time but without any luck. Google has detected somewhere in my website harmful content (Malware). But the problem is, I could not find ...
0
votes
0answers
19 views

Angular2 - Google indexing loader instead of redirected home page

I'm working on a multilanguage app in Angular. Universal is already setup and working but I have issue with googlebot indexing the page. When user navigates to url https://example.com only loader ...
4
votes
1answer
42 views

Does Google return to visit a page with 404 Error?

If a page indexed in Google shows a 404 Error, does Google return to visit it again? In other words, if a page that was indexed in Google continue to show the 404 Error for several years, will ...
3
votes
1answer
27 views

Status code vs. Response Header Status: what is the rule for bots?

Assuming we have a page, which has as Status Code 200 OK and as Response Headers Status HTTP/1.1 410 Gone, like on image: What is the behavior of bots / Googlebot? To crawl/index or not?
1
vote
1answer
39 views

Googlebot refuses to render more than index.html in Ember app

I am using Ember 3.0.0 to create a booking site. Title and meta tags are handled via ember-cli-meta-tags and seem to be rendered correctly when pages are opened in the web browser. However, Googlebot ...
1
vote
2answers
41 views

Does Google's own bot contribute to my Google Maps API fees?

Does Googlebot, when crawling pages which display maps using the Javascript API, cause a billable API usage on each page request?
0
votes
2answers
63 views

Stop Google from accsesing pages without robots.txt

Is it possible to get Google to stop crawling certain pages without using robots.txt? My understanding is robots.txt can block Google from accsesing a page, but if using noindex (meta or header) ...
1
vote
0answers
28 views

Could using the Google URL removal tool for a specific page also have removed the home page?

I noticed today my Homepage URL is not on Google Search Result anymore. Firstly for specific keyword where i ranked #1 than for domain itself. it is my brand keyword which I get most traffic from. I ...
0
votes
0answers
9 views

Google Search is displaying the wrong snippet [duplicate]

If you search for my website in google you'll get 3 results, one of them works as requested, right title right snippet description, on has only the right title and one only the right description. I ...
1
vote
1answer
54 views

Sitewide A/B or Sanity Test

We are switching to a new Wordpress template and need some help. We will monitor if the test version improves the sites performance (CTR, Revenue) Technically we will: Split A Records by 50% for ...
0
votes
1answer
14 views

Does Google take into account page title changes due to JavaScript? [duplicate]

One of my pages is set up so that the initial page title is just some placeholder title, and it changes after an AJAX call. Here's what the process looks like: Initial page load AJAX call to ...
1
vote
0answers
17 views

Stop getting traffic from Googlebot on high server load

In my website, we usually get a spike in the number of requests because of search engine crawlers (like Googlebot) harvesting the website for indexing. So what I thought of; is depending on the ...
1
vote
0answers
40 views

Google AdWords ads are disapproved because “destination not working”

We have been having an issue setting up adwords for one of our clients and despite multiple phone calls to Google we cannot get an answer out of them as to why our ads are being disapproved. ...
0
votes
0answers
18 views

How to manage multi-city or multi-area content within a country? [duplicate]

We have a gold jewellery website with 1000 products available in 500 cities. Based on the city- the price, offers, and physical stores are different for each product. Currently, we have 1 webpage ...
1
vote
1answer
39 views

SEO impact of changing versions (and names) of external CSS

A website of mine uses a WordPress plugin for CSS minification. The plugin regenerates the CSS file on every update to the style-sheet. When it does, the plugin also changes the version and hence name ...
0
votes
0answers
10 views

Relative URLs vs absolute URLs for SEO [duplicate]

This question has some fantastic answers on StackOverflow, especially from the dev perspective like this one https://stackoverflow.com/a/21828923/728236, but how about from an SEO perspective? There'...
1
vote
0answers
18 views

Product carousel on my site is showing up blank in Google's fetch and render for mobile

Using GSC, I am trying to fetch a page with products and content, this page also has few products in carousel format and when I render this page for desktop or mobile, I am seeing everything fine but ...
1
vote
0answers
13 views

What is the correct response to Googlebot for a search_term_string from site search markup with _escaped_fragment?

I have a AJAX web online-shop with hashbang URL and added SiteLinks as JSON-LD. I follow this introduction. { "@context":"http://schema.org", "@type":"WebSite", "url":"https://...
0
votes
0answers
17 views

Browser response vs crawler response: different HTML length

Because of length, Google and Yandex bots are saying to me that the page is not the same and that you are cloaking. But I do nothing about cloaking, here is the diff: Browser: <div class="...
0
votes
0answers
31 views

Seo Report Result Problems

This is my first seo project.I am new at SEO.Have some problems, Info : Webform ASP.NET 4.5 1-) "Accesible Index Page Configured wrong" error is my biggest problem.My site have 4 URL for homepage . ...
1
vote
0answers
10 views

My most important pages does not get indexed, google search console [duplicate]

I have a website with blog articles. I added it's sitemap to google search console and it say that the sitemap has been processed. However, half of my articles (the important pages) have not been ...
1
vote
0answers
11 views

Should a page with adsense be viewed at least once for adsense crawler to crawl?

I use plugins to insert Adsense codes on my WordPress website. The plugin has an option to disable ads for the admin user. Hence, I will not be seeing ads when creating and publishing/reviewing posts. ...
0
votes
0answers
13 views

header tags offscreen - punished by google? [duplicate]

If I put a few h2 and h3 tags with seo phrases on my site then used css or js to move them off the visible screen would googlebot be able to see that and would they punish me for doing so? There are ...
1
vote
0answers
33 views

Google cache is not showing up for a page despite trying twice to use the fetch and render tool [duplicate]

I did Fetch and Render twice on a particular page but the cached version of the page is still not showing up on Google. I did a Fetch and Render on other posts on the site and they all showed cached ...
0
votes
0answers
18 views

Any relation between adsense lack of impressions and incorrect sitemap?

I have recently submitted a sitemap in search console for at least 200 new pages on my site. However, I did a mistake by not including a slash ("/") at the end of each url. But the pages without ...
1
vote
1answer
47 views

CloudFront - disable access to distribution via d1234.cloudfront.net

I have a CloudFront distribution with a configured alternative name (e.g. cdn.example.com). Is it possible to restrict access to the CDN via d1234.cloudfront.net name? I don't want it to be indexed ...
1
vote
1answer
91 views

Google intermittently fails to download CSS and JS from CloudFront CDN

We noticed a lot of "Mobile Usability" errors in the Search Console. When I clicked on them and chose "Check live version" option - it always said the page was mobile friendly. However, the mobile ...
0
votes
2answers
57 views

Status Code 500 to Googlebot but url working

I have a URL on my site that is working perfectly. I have the robots with the index and follow, but when I Fetch as Google in the WMT says Unreachable and gives me this message in the crawl errors ...
0
votes
0answers
16 views

Why google index the pages that has been blocked on robots.txt? [duplicate]

I found this very interesting. I already blocked those pages via robots.txt but it keeps on showing on my Search Console for Smartphone. Does adding this: User-agent: Googlebot-mobile Allow: /*?...
3
votes
0answers
39 views

Is it necessary to block googlebot and googlebot-video to block video crawl?

We have problem with declining CTR caused by unwanted video snippet on SERP. Therefore we have taken measure to block the Googlebot from crawling our video file. User-Agent: Googlebot Disallow: /*....
4
votes
1answer
117 views

Dynamic content using .append() or .innerHTML() and SEO

I was looking for some clarification on something I am looking to implement, I would like to know how a Google Crawl will see this and if it would be detrimental to SEO. On my webpage I have an empty ...
4
votes
1answer
52 views

Does Google follow and count a backlink that goes through a JavaScript location.href redirect page?

example.com has a link on their website which is on same domain like, <a href="https://example.com?ref=mywebsite.tld">Reference</a> example.com?ref=mywebsite.tld doesn't redirect with ...
0
votes
0answers
10 views

Removing Paginated URLs from Google Search [duplicate]

I have a website running on Django, Python. I'm using django pagination so that users can click next & see further content. For example, my website is example.com, so when user requests new page ...
1
vote
0answers
23 views

Google showing Pages with `noindex` meta tag [duplicate]

I have a website running on Django, Python. I have 2 simple questions, First: A week ago I made few changes in the main urls of website. For example https://example.com/user/account/profile is ...
0
votes
0answers
9 views

How to make Google show my blog page which is on a different domain?

I have a website (eg. https://work.example.com ) which is being crawled by google. I have a blog for the same under https://blogs.example.com. When I search in google with keywords like "work example",...
2
votes
0answers
68 views

Does Googlebot cache JavaScript files between two pages if they expire in one year?

My question is about javascript websites. Suppose GoogleBot has crawled example.com/index.html?id=1 Does it request all the javascript files again when crawling example.com/index.html?id=2? index....
1
vote
0answers
15 views

which status code should be returned for null parameter

having a website with many urls in js code (ajax calls), the google bot is calling these but without the necessary parameters atm 200 code is returned and the html for the error page, and this is ...
1
vote
2answers
54 views

How to get Googlebot to crawl my website faster to pick up my new meta description

I've tried to put an new meta description for my website, but when I look it up I get the wrong description. After some research I found out that I need Google to crawl my site. The cache:website ...
0
votes
0answers
9 views

URL not crawled to Google webmaster [duplicate]

This is the website sitemap url - https://www.geeks.sydney/sitemap.xml. I have submitted this url to Google search console/webmaster about 6-8 months ago. Still now I can see that, 171 url are indexed ...
4
votes
1answer
44 views

AdSense crawler access fails because the bot adds a slash to the login URL

I'm trying to get a site verified for AdSense. Since I want to display ads in authenticated pages too, I'm setting up Crawler Access to give a username and password to Google crawler. In this site, ...
3
votes
1answer
98 views

Why would googlebot be trying to crawl a URL “https://www.example.com/https://www.example.com/other/things”?

I've been trying to clean up crawl errors on my company's website. In one menu, google bot is trying to crawl every link with an extra-base URL attached. These pages (perhaps obviously) don't exist ...
1
vote
1answer
22 views

How to get a live-steam video into the carousel at the top of the Google search results?

I want whenever a live stream content is available on my website like a sports match then I want to Google to show a live stream preview in its result from the video tag. What I think is I need to ...
3
votes
2answers
115 views

Googlebot says access denied on non-existant resource

Google Search Console is telling me: Access Denied Googlebot couldn't crawl your URL because your server either requires authentication to access the page, or it is blocking Googlebot from ...
-1
votes
1answer
69 views

Maximum time to cache HTTPS URL after HTTP to HTTPS migration?

We have migrated our site from HTTP to HTTPS. How much time will the Google crawler take to cache all our HTTPS pages. We have more than 50K pages in our website.
1
vote
1answer
39 views

Googlebot smartphone redirect

We are using two different domains for our desktop and mobile websites. "www.x.com" and "m.x.com". The users with mobile devices ending on "www.x.com" are (302) redirected to "m.x.com" I have just ...