Stack Exchange Network

Stack Exchange network consists of 174 Q&A communities including Stack Overflow, the largest, most trusted online community for developers to learn, share their knowledge, and build their careers.

Visit Stack Exchange

A computer program that accesses web pages for various purposes (to scrape content, to provide search engines with information about your site, etc.)

0
votes
1answer
15 views

SEO - Custom error pages & status codes

My website uses a custom error page with the <meta name="robots" content="noindex"> meta tag. Am I right in thinking this tag will tell crawlers, such as Googlebot, to remove the page from ...
0
votes
0answers
30 views

Robots are still Crawling Website after Disallowing Using robot.txt and Redirecting it Https from Http using .htaccess [closed]

I have tried to disallow robots to crawling my website by adding robot.txt and redirecting website to https as Explained below .htacces Contains the following code:- RewriteEngine on RewriteCond %{...
1
vote
0answers
10 views

how do i search for certain word in a website? [closed]

Is there any way to search within a website without manually checking out every link? Suppose there is some website www.somewebsite.com and I want to search for a word in all of the pages www....
0
votes
2answers
40 views

Major Crawlers not honoring robots.txt?

few of my websites were compromised, there were thousands of links submitted to google, bing etc which are not relevant to my site. I cleaned the websites in question and updated the robots.txt. the ...
1
vote
1answer
23 views

Alternative languages redirecting

Serving a page in English, EG: https://www.example.com/en/make-games Has the following meta tags: <meta rel="alternate" href="https://www.example.com/fr/make-games" hreflang="fr" /> <meta ...
-1
votes
0answers
11 views

Why google is caching site without www [duplicate]

Greeting of the day! My Question is that why google is cacahing the website without www. let me explain in brief when i check the webcache with www it showing 404 error on google check link below ...
2
votes
2answers
37 views

Ampersand (&) in actual URL and sitemap

I have a sitemap (which is submitted to Google for indexing) in which I am adding URLs which have an ampersand (&) in it. Since in sitemap & has to be escaped I replaced & with &amp; ...
3
votes
1answer
51 views

How to hide HTML tags from SEO spiders

I understand and know how hiding things from your website works and how the search engine spiders will still be able to ¨see¨ this hidden element and penalize your website for it. Thing is, I did not ...
1
vote
1answer
42 views

Google started to reports root url blocked by robots.txt - but it's not -

My website has been running for multiple years without problem, however I recently noticed my root url on Google does not show any description/title. The Webmaster Tool reports this error: Crawl ...
3
votes
1answer
65 views

Should I publish an empty ads.txt file for a site that does not run any ads?

I have seen a few 404 errors when ads.txt is requested on some of the sites that I manage. I Guess the answer may be identical to the question about an empty robots.txt file or no file at all (just ...
0
votes
0answers
21 views

How to block RiskIQ crawlers?

RiskIQ is kind of "private Google" which crawls and index entire Internet. Unlike Google: they do not respect robots.txt, so there is no opt-out nor rate-limiting. they use User-Agent of normal ...
4
votes
1answer
54 views

Does Google return to visit a page with 404 Error?

If a page indexed in Google shows a 404 Error, does Google return to visit it again? In other words, if a page that was indexed in Google continue to show the 404 Error for several years, will ...
0
votes
1answer
18 views

How deep do search engine crawlers go?

I have 2 sites with very similar HTML structure and I would like to know how google or any other crawler rank these 2 sites. Will they penalize me due the sites sharing a structure? The content ...
1
vote
1answer
46 views

Googlebot refuses to render more than index.html in Ember app

I am using Ember 3.0.0 to create a booking site. Title and meta tags are handled via ember-cli-meta-tags and seem to be rendered correctly when pages are opened in the web browser. However, Googlebot ...
0
votes
2answers
71 views

Stop Google from accsesing pages without robots.txt

Is it possible to get Google to stop crawling certain pages without using robots.txt? My understanding is robots.txt can block Google from accsesing a page, but if using noindex (meta or header) ...
1
vote
0answers
24 views

Stop getting traffic from Googlebot on high server load

In my website, we usually get a spike in the number of requests because of search engine crawlers (like Googlebot) harvesting the website for indexing. So what I thought of; is depending on the ...
1
vote
0answers
136 views

What is user agent ;__CT_JOB_ID__:?

I've been searching the web for an explanation of the sudden influx we've gotten of user agents of this format: ;__CT_JOB_ID__:68a72eee-353c-4b0b-82ae-0cb6c8ca3e6e; Each visitor has a unique IP. So ...
1
vote
0answers
19 views

Product carousel on my site is showing up blank in Google's fetch and render for mobile

Using GSC, I am trying to fetch a page with products and content, this page also has few products in carousel format and when I render this page for desktop or mobile, I am seeing everything fine but ...
2
votes
3answers
57 views

Is it better to start a website with a new domain or an existing domain that had downtime?

For the past year I have been working on a website online and offline. I would take the website down for weeks or months and work on it locally instead. Now that I am fully ready to launch it, is it ...
1
vote
0answers
12 views

Should a page with adsense be viewed at least once for adsense crawler to crawl?

I use plugins to insert Adsense codes on my WordPress website. The plugin has an option to disable ads for the admin user. Hence, I will not be seeing ads when creating and publishing/reviewing posts. ...
1
vote
2answers
38 views

MEAN stack app robots.txt not performing as expected

I am trying to use robots.txt to allow crawling public links on my SPA, but to disallow other links to the site. When I use the robots.txt tester by Google, it does not differentiate between the two ...
0
votes
1answer
29 views

Should I block web crawlers with robots.txt while my site is under maintenance?

I've recently inherited a quite popular website (fan page for a band), but the old webmaster had to take it down because of GDPR issues. There was a lot of content available on the website. I plan to ...
0
votes
0answers
17 views

Google Deindexing all our Pages - Strange Issue [duplicate]

We're facing deindexing issues for so many pages since two weeks and have lost rankings and traffic. When we check the URLs with inspection tool in webmaster, the canonical URLs are declared as ...
1
vote
1answer
138 views

Google intermittently fails to download CSS and JS from CloudFront CDN

We noticed a lot of "Mobile Usability" errors in the Search Console. When I clicked on them and chose "Check live version" option - it always said the page was mobile friendly. However, the mobile ...
1
vote
1answer
178 views

Google Search Console Crawl 404 Error for some wp-content directories even when those directories contain files

Here is a picture of my Google Search Console: When I try to click at the url for example error 18: wp-content/uploads/2016/01/ Appear File not found. Of course inside the path (wp-content/uploads/...
4
votes
1answer
155 views

Dynamic content using .append() or .innerHTML() and SEO

I was looking for some clarification on something I am looking to implement, I would like to know how a Google Crawl will see this and if it would be detrimental to SEO. On my webpage I have an empty ...
2
votes
1answer
106 views

Can Angular project be indexed in Google without Angular Universal?

I'm working on an Ionic PWAprogressive web app (PWA) and forthere are reasons iI cannot use Angular Universal. Is there any way to index with success inget Google to successfully index the site ...
2
votes
1answer
99 views

Is this site cloaking to hide my backlink from spiders?

I wrote a guest post at the following URL: https://wparena.com/integrate-google-tag-manager-wordpress/ At the end, you can see my author box with a URL pointing back to my site. This link is a ...
2
votes
0answers
14 views

Prevent bad rankings on a Q and A site for questions without an answer by using noindex until answered [duplicate]

I have a website on which users can ask a question & also they can answer it. So, when a user asks a question, it creates a new web-page on the website & below that question lies it's answers ...
1
vote
0answers
29 views

Do any search engines exist which disobey rel=“nofollow” or “Disallow”? [closed]

Moving beyond the big players - Google, Yahoo, Bing, Ask, DDG (who all obey "nofollow"s), are there any search engines out there which deliberately disobey rel='nofollow' links and Disallow rules in ...
0
votes
0answers
10 views

Removing Paginated URLs from Google Search [duplicate]

I have a website running on Django, Python. I'm using django pagination so that users can click next & see further content. For example, my website is example.com, so when user requests new page ...
1
vote
1answer
26 views

Another url is indexed in google which has a link ending with rpid=5PO84Q269

My Website is indexed in google. When i was searching for the term in google for my website i found a url placing above my websites root URL which has same url as root URL but has a weird url ...
1
vote
0answers
25 views

Google showing Pages with `noindex` meta tag [duplicate]

I have a website running on Django, Python. I have 2 simple questions, First: A week ago I made few changes in the main urls of website. For example https://example.com/user/account/profile is ...
0
votes
0answers
9 views

How to make Google show my blog page which is on a different domain?

I have a website (eg. https://work.example.com ) which is being crawled by google. I have a blog for the same under https://blogs.example.com. When I search in google with keywords like "work example",...
2
votes
0answers
87 views

Does Googlebot cache JavaScript files between two pages if they expire in one year?

My question is about javascript websites. Suppose GoogleBot has crawled example.com/index.html?id=1 Does it request all the javascript files again when crawling example.com/index.html?id=2? index....
3
votes
2answers
212 views

Is the top entry and exit page in Webalizer reports the most popular page on my site?

According to Webalizer, the top entry and exit page to my website is "/blog/". This is where my Wordpress blog is situated. (There are static non-Wordpress pages on my site as well.) Does that mean ...
2
votes
0answers
42 views

Crawling a SalesForce site fully generated by JS with Screaming Frog only gets content on some pages

I'm trying to crawl a Salesforce-generated site. This site appears to be fully generated by Javascript (if you load the page with JS disabled only a white screen shows up). For instance, Salesforce's ...
3
votes
3answers
265 views

Blocking bad bots in .htacess - should I add a BLOCK list or ALLOW list?

I got my Siteground hosting account suspended yesterday due to the allowed CPU overage. Besides recommendation to disable or control the WP heartbeat API, I was also recommended to block bad bots and ...
0
votes
0answers
9 views

URL not crawled to Google webmaster [duplicate]

This is the website sitemap url - https://www.geeks.sydney/sitemap.xml. I have submitted this url to Google search console/webmaster about 6-8 months ago. Still now I can see that, 171 url are indexed ...
5
votes
1answer
56 views

AdSense crawler access fails because the bot adds a slash to the login URL

I'm trying to get a site verified for AdSense. Since I want to display ads in authenticated pages too, I'm setting up Crawler Access to give a username and password to Google crawler. In this site, ...
1
vote
2answers
98 views

Delivering a different version of website to Google robots (SSR)

I am working on an enterprise website project as an SEO manager. They were using Angular.js by now but I made them switch to SSR (server-side rendering) since Google robots were not able to render ...
2
votes
0answers
106 views

Which webmail providers spider links in emails? [closed]

Does anyone know which webmail providers spider links in emails? I know that Outlook.com does, because I made a canary token and emailed it to an Outlook email address and the token got crawled ...
2
votes
1answer
54 views

How to make Google display only Country-specific search results based on user's location

My question goes like this: I own a multinational company with the following websites www.abc.com/us/ www.abc.com/uk/ www.abc.com/apac/ I advertise my product (let's name it "device") on each of my ...
-1
votes
1answer
70 views

Maximum time to cache HTTPS URL after HTTP to HTTPS migration?

We have migrated our site from HTTP to HTTPS. How much time will the Google crawler take to cache all our HTTPS pages. We have more than 50K pages in our website.
0
votes
1answer
40 views

How to identify all search engine bots and crawlers?

I'm recently doing keyword advertising on Bing, which also powers AOL and Yahoo searches, to promote a product. And I'm using a third-party tracker to track the clicks of my advertisements. I found ...
1
vote
0answers
13 views

crawling issue in my website about two week [duplicate]

I almost did this change before 20-21 days but still Google is not crawling Site.
0
votes
1answer
97 views

No page information in search results | Magento 2.1.8

When I google my domain, I can find it to be first but it's saying "No information is available for this page" The website has meta information like this: <meta name="description" content="My ...
1
vote
1answer
31 views

SEO crawler is reporting an issues with a high number of external links, is this a problem?

Deep Crawl shows an issue of high external linking (68) for my website. I thought external linking is good for the website! Is this an issue that needs to be fixed to improve SEO, and if so, what is ...
1
vote
2answers
56 views

Would multiple images with the same alt text as the page name be keyword stuffing?

In a page I have three or four images that have same alt tag as of my page name. Could Google consider that to be keyword and penalize my site? Is it good practice to include page name in alt tag? ...
1
vote
4answers
193 views

Big discrepancy in number of known pages in Google Search Console and ScreamingFrog

In a technical SEO audit I used both Google Search Console and ScreamingFrog crawl to review the status errors, however there is big discrepancy in the number of pages that Google knows about, and the ...