Stack Exchange Network

Stack Exchange network consists of 174 Q&A communities including Stack Overflow, the largest, most trusted online community for developers to learn, share their knowledge, and build their careers.

Visit Stack Exchange

Questions tagged [web-crawlers]

A computer program that accesses web pages for various purposes (to scrape content, to provide search engines with information about your site, etc.)

1
vote
1answer
22 views

How to add robots.txt to my WordPress site?

How to add robots.txt to my wordpress.com site?
1
vote
1answer
20 views

Should we Put Coupon Pages into sitemap because they do have a faster expiry rate?

Is it a good practice to keep coupons/ voucher pages into a sitemap. because they do have a faster expiry rate. Case: We are updating our sitemap once in 24 hours so, will it be a good practice to ...
2
votes
2answers
25 views

How many levels of sub-pages do Google bots crawl to when fetching and indexing linked pages in Google Search Console?

Let's say I submit a sitemap page that contains all the links I want Google to crawl, but instead of submitting a sitemap, I fetch the page and linked pages and let Google index them. How many levels ...
1
vote
1answer
22 views

Can Google crawl .TS (MPEG2-TS) video format?

We are using the tool mp4hls (https://www.bento4.com/developers/hls/) to prep the hls stream. Basically, I think it can conver any video format to .mp4 then it convert to .ts. However, Google only ...
1
vote
1answer
39 views

SPA website, would making an HTML-only version for SEO be worthy?

I made a Vue.js SPA, and since it could affect SEO (specially because a large portion of my client base is not using Google), I was thinking: why not providing an HTML-only version for it? Would ...
0
votes
0answers
10 views

noindex for a piece of a text [duplicate]

I have a piece of text in the homepage of one of my websites that I don't want Google to scan at all --- I don't want it to effect the SEO in any way. I can upload it as an image but it's not a good ...
0
votes
1answer
29 views

Error in search console!

the new search console is pointing out usability errors on mobile devices on wordpress upload pages, that is, pages where I can not change font size, viewport window and things like that, how do I ...
1
vote
1answer
30 views

Google Bot Crawls using expired cookies and fails to get response

We are using a cookie on our website which is to fetch some information through an API. The cookie becomes invalid after a very small duration. Google Bot uses the same cookie again and again to ...
2
votes
1answer
78 views

htaccess 410 url spam hack requests excluding site generated request

Our site was hacked and links to random content where added to the site. We completely removed the hacked site and put a new one in its place. Everything new including images and content, no other ...
1
vote
2answers
158 views

Deleted tags & categories from my blog still come up in Google search results

I am a newbie in blogging. I've deleted some of the unwanted tags and categories from my wordpress blog but they kinda still come put in the search results when I search my website on Google. How do ...
4
votes
0answers
57 views

Will forbidding “bad bot” user agents via .htaccess add a layer of protection against email address scraping beyond the existing captcha?

I was reviewing the security/protection measures for the websites I manage (one is for a small consulting firm, the other for a not-for-profit art association), and I came across the topic of bad-bot ...
2
votes
0answers
29 views

Can many single page hits from a variety user agents from the same VPN with the same screen resolution be hurting my search rankings?

I am trying to clarify what is happening with the traffic on my site. Lately I am being flooded with many strange hits, mostly from US and they seem to only open 1 page or visit for few seconds. The ...
1
vote
1answer
19 views

SEO - Custom error pages & status codes

My website uses a custom error page with the <meta name="robots" content="noindex"> meta tag. Am I right in thinking this tag will tell crawlers, such as Googlebot, to remove the page from ...
1
vote
0answers
10 views

how do i search for certain word in a website? [closed]

Is there any way to search within a website without manually checking out every link? Suppose there is some website www.somewebsite.com and I want to search for a word in all of the pages www....
0
votes
2answers
44 views

Major Crawlers not honoring robots.txt?

few of my websites were compromised, there were thousands of links submitted to google, bing etc which are not relevant to my site. I cleaned the websites in question and updated the robots.txt. the ...
1
vote
1answer
23 views

Alternative languages redirecting

Serving a page in English, EG: https://www.example.com/en/make-games Has the following meta tags: <meta rel="alternate" href="https://www.example.com/fr/make-games" hreflang="fr" /> <meta ...
2
votes
2answers
62 views

Ampersand (&) in actual URL and sitemap

I have a sitemap (which is submitted to Google for indexing) in which I am adding URLs which have an ampersand (&) in it. Since in sitemap & has to be escaped I replaced & with &amp; ...
3
votes
1answer
66 views

How to hide HTML tags from SEO spiders

I understand and know how hiding things from your website works and how the search engine spiders will still be able to ¨see¨ this hidden element and penalize your website for it. Thing is, I did not ...
1
vote
1answer
83 views

Google started to reports root url blocked by robots.txt - but it's not -

My website has been running for multiple years without problem, however I recently noticed my root url on Google does not show any description/title. The Webmaster Tool reports this error: Crawl ...
3
votes
1answer
69 views

Should I publish an empty ads.txt file for a site that does not run any ads?

I have seen a few 404 errors when ads.txt is requested on some of the sites that I manage. I Guess the answer may be identical to the question about an empty robots.txt file or no file at all (just ...
4
votes
1answer
69 views

Does Google return to visit a page with 404 Error?

If a page indexed in Google shows a 404 Error, does Google return to visit it again? In other words, if a page that was indexed in Google continue to show the 404 Error for several years, will ...
0
votes
1answer
18 views

How deep do search engine crawlers go?

I have 2 sites with very similar HTML structure and I would like to know how google or any other crawler rank these 2 sites. Will they penalize me due the sites sharing a structure? The content ...
1
vote
1answer
55 views

Googlebot refuses to render more than index.html in Ember app

I am using Ember 3.0.0 to create a booking site. Title and meta tags are handled via ember-cli-meta-tags and seem to be rendered correctly when pages are opened in the web browser. However, Googlebot ...
0
votes
2answers
72 views

Stop Google from accsesing pages without robots.txt

Is it possible to get Google to stop crawling certain pages without using robots.txt? My understanding is robots.txt can block Google from accsesing a page, but if using noindex (meta or header) ...
1
vote
0answers
39 views

Stop getting traffic from Googlebot on high server load

In my website, we usually get a spike in the number of requests because of search engine crawlers (like Googlebot) harvesting the website for indexing. So what I thought of; is depending on the ...
1
vote
0answers
226 views

What is user agent ;__CT_JOB_ID__:?

I've been searching the web for an explanation of the sudden influx we've gotten of user agents of this format: ;__CT_JOB_ID__:68a72eee-353c-4b0b-82ae-0cb6c8ca3e6e; Each visitor has a unique IP. So ...
1
vote
0answers
19 views

Product carousel on my site is showing up blank in Google's fetch and render for mobile

Using GSC, I am trying to fetch a page with products and content, this page also has few products in carousel format and when I render this page for desktop or mobile, I am seeing everything fine but ...
2
votes
3answers
59 views

Is it better to start a website with a new domain or an existing domain that had downtime?

For the past year I have been working on a website online and offline. I would take the website down for weeks or months and work on it locally instead. Now that I am fully ready to launch it, is it ...
1
vote
0answers
12 views

Should a page with adsense be viewed at least once for adsense crawler to crawl?

I use plugins to insert Adsense codes on my WordPress website. The plugin has an option to disable ads for the admin user. Hence, I will not be seeing ads when creating and publishing/reviewing posts. ...
1
vote
2answers
44 views

MEAN stack app robots.txt not performing as expected

I am trying to use robots.txt to allow crawling public links on my SPA, but to disallow other links to the site. When I use the robots.txt tester by Google, it does not differentiate between the two ...
0
votes
1answer
29 views

Should I block web crawlers with robots.txt while my site is under maintenance?

I've recently inherited a quite popular website (fan page for a band), but the old webmaster had to take it down because of GDPR issues. There was a lot of content available on the website. I plan to ...
0
votes
0answers
18 views

Google Deindexing all our Pages - Strange Issue [duplicate]

We're facing deindexing issues for so many pages since two weeks and have lost rankings and traffic. When we check the URLs with inspection tool in webmaster, the canonical URLs are declared as ...
2
votes
1answer
204 views

Google intermittently fails to download CSS and JS from CloudFront CDN

We noticed a lot of "Mobile Usability" errors in the Search Console. When I clicked on them and chose "Check live version" option - it always said the page was mobile friendly. However, the mobile ...
1
vote
1answer
291 views

Google Search Console Crawl 404 Error for some wp-content directories even when those directories contain files

Here is a picture of my Google Search Console: When I try to click at the url for example error 18: wp-content/uploads/2016/01/ Appear File not found. Of course inside the path (wp-content/uploads/...
4
votes
1answer
192 views

Dynamic content using .append() or .innerHTML() and SEO

I was looking for some clarification on something I am looking to implement, I would like to know how a Google Crawl will see this and if it would be detrimental to SEO. On my webpage I have an empty ...
2
votes
1answer
262 views

Can Angular project be indexed in Google without Angular Universal?

I'm working on an Ionic PWAprogressive web app (PWA) and forthere are reasons iI cannot use Angular Universal. Is there any way to index with success inget Google to successfully index the site ...
2
votes
1answer
104 views

Is this site cloaking to hide my backlink from spiders?

I wrote a guest post at the following URL: https://wparena.com/integrate-google-tag-manager-wordpress/ At the end, you can see my author box with a URL pointing back to my site. This link is a ...
2
votes
0answers
14 views

Prevent bad rankings on a Q and A site for questions without an answer by using noindex until answered [duplicate]

I have a website on which users can ask a question & also they can answer it. So, when a user asks a question, it creates a new web-page on the website & below that question lies it's answers ...
1
vote
0answers
31 views

Do any search engines exist which disobey rel=“nofollow” or “Disallow”? [closed]

Moving beyond the big players - Google, Yahoo, Bing, Ask, DDG (who all obey "nofollow"s), are there any search engines out there which deliberately disobey rel='nofollow' links and Disallow rules in ...
0
votes
0answers
10 views

Removing Paginated URLs from Google Search [duplicate]

I have a website running on Django, Python. I'm using django pagination so that users can click next & see further content. For example, my website is example.com, so when user requests new page ...
1
vote
1answer
27 views

Another url is indexed in google which has a link ending with rpid=5PO84Q269

My Website is indexed in google. When i was searching for the term in google for my website i found a url placing above my websites root URL which has same url as root URL but has a weird url ...
1
vote
0answers
34 views

Google showing Pages with `noindex` meta tag [duplicate]

I have a website running on Django, Python. I have 2 simple questions, First: A week ago I made few changes in the main urls of website. For example https://example.com/user/account/profile is ...
0
votes
0answers
9 views

How to make Google show my blog page which is on a different domain?

I have a website (eg. https://work.example.com ) which is being crawled by google. I have a blog for the same under https://blogs.example.com. When I search in google with keywords like "work example",...
2
votes
0answers
113 views

Does Googlebot cache JavaScript files between two pages if they expire in one year?

My question is about javascript websites. Suppose GoogleBot has crawled example.com/index.html?id=1 Does it request all the javascript files again when crawling example.com/index.html?id=2? index....
3
votes
2answers
216 views

Is the top entry and exit page in Webalizer reports the most popular page on my site?

According to Webalizer, the top entry and exit page to my website is "/blog/". This is where my Wordpress blog is situated. (There are static non-Wordpress pages on my site as well.) Does that mean ...
2
votes
0answers
50 views

Crawling a SalesForce site fully generated by JS with Screaming Frog only gets content on some pages

I'm trying to crawl a Salesforce-generated site. This site appears to be fully generated by Javascript (if you load the page with JS disabled only a white screen shows up). For instance, Salesforce's ...
3
votes
3answers
348 views

Blocking bad bots in .htacess - should I add a BLOCK list or ALLOW list?

I got my Siteground hosting account suspended yesterday due to the allowed CPU overage. Besides recommendation to disable or control the WP heartbeat API, I was also recommended to block bad bots and ...
0
votes
0answers
9 views

URL not crawled to Google webmaster [duplicate]

This is the website sitemap url - https://www.geeks.sydney/sitemap.xml. I have submitted this url to Google search console/webmaster about 6-8 months ago. Still now I can see that, 171 url are indexed ...
5
votes
1answer
75 views

AdSense crawler access fails because the bot adds a slash to the login URL

I'm trying to get a site verified for AdSense. Since I want to display ads in authenticated pages too, I'm setting up Crawler Access to give a username and password to Google crawler. In this site, ...
1
vote
2answers
129 views

Delivering a different version of website to Google robots (SSR)

I am working on an enterprise website project as an SEO manager. They were using Angular.js by now but I made them switch to SSR (server-side rendering) since Google robots were not able to render ...