Skip to main content

Questions tagged [web-crawlers]

A computer program that accesses web pages for various purposes (to scrape content, to provide search engines with information about your site, etc.)

Filter by
Sorted by
Tagged with
1 vote
1 answer
40 views

Do web crawlers clear their browser cache?

Suppose you make a website and you forget to set the proper cache control headers, and your index.html is set to be cached and never expire. You fix this, and after clearing your browser cache this ...
Mystical's user avatar
  • 113
0 votes
1 answer
357 views

How do I stop Amazon from crawling a website?

I would like to prevent Amazon from scraping product data on my website. So I found this document: https://developer.amazon.com/amazonbot And this example: User-agent: Amazonbot # Amazon's ...
user avatar
0 votes
0 answers
12 views

Why aren't search engines showing my site anymore? [duplicate]

I have created a site as a home project but for some reason, it stopped showing my site in Google. I connected my site to Google Search Console and most of the pages are indexed. At least I can see it ...
Sporx's user avatar
  • 1
0 votes
0 answers
17 views

Weird Apache requests

I've recently migrated a client from a WIX site to a Wordpress site (running on an Apache server) I am seeing quite a few requests in the apache log file in the format site.name:443 2.2.2.2 - - [19/...
davidgo's user avatar
  • 8,071
3 votes
1 answer
205 views

Google Search Console: My "indexed" and "not-indexed" pages have flipped. I now have more "not-indexed" pages

Web App Google Search Console Index: Crossover I have a web app with a lot of dynamically generated pages. Each page has similar html layout with different text/image content, as well as a unique url. ...
Jordan Babb's user avatar
0 votes
0 answers
21 views

Exceed crawl limits for a website [duplicate]

In previous month i created a lot of pages (> 200K), and google indexed it. After some time a relized that i don't need some of these pages and created a new ones, but google don't indexing it. I ...
Kenix's user avatar
  • 23
2 votes
1 answer
46 views

JSON Files getting crawled suddenly

Just noticed that GSC has recently taken an interest in the .json files that are associated to all of the product pages on Shopify, and is now crawling them, but wasn't before. Should I be worried ...
Will's user avatar
  • 21
0 votes
0 answers
30 views

How does Serpi protect from web crawling restrictions

I'm thinking of making a website to find the lowest price of an item on the net. The problem is that in the terms of use of Google search API it says that it is forbidden to: Scrape, build databases, ...
Jenia Ivanov's user avatar
1 vote
0 answers
257 views

Should Feed/RSS Be Blocked from Robots.txt?

Should Feed/RSS be blocked via robots.txt? Disallow: /*feed* Disallow: /*rss* If yes, is this the correct way to apply? If you are sending Feed/RSS somewhere or if you have a subscription system, of ...
gfo95's user avatar
  • 41
4 votes
1 answer
46 views

How to protect from spam when the legislation forces you to display the email address?

Context I want to protect a ecommerce website from email spam, whilst the legislation in my country makes mandatory to publish an email contact address on ecommerce sites. As an additional constraint, ...
OuzoPower's user avatar
  • 357
3 votes
1 answer
273 views

SEMRush Site Audit can't crawl PDFs on my site, is this an issue with my server?

My agency uses SEMRush, and we especially find the Site Audit function helpful for spotting potential issues with our SEO and UX. However of late, SEMRush has been flagging PDFs quite frequently, ...
GeoffAtkins's user avatar
  • 2,277
1 vote
1 answer
38 views

What is the Query to add in robots.txt for blocking URLs has some special characters?

I was going through the Google Search Console pages issues and one of the issue reason is "Alternate page with proper canonical tag" and this has some URLs which look like below example.com/...
Rahul Chalana's user avatar
0 votes
1 answer
248 views

Use passive listeners to improve scrolling performance

Google's Lighthouse tests for CWV ran on my website include an issue stating that touch and wheel event listeners are not marked as passive. How and where to fix this? I have tried to check my file ...
Chek Reynaldo's user avatar
1 vote
0 answers
128 views

Connecting to MySQL database hosted on CPanel

I have followed all of the usual steps in setting up a pipeline but cannot seem to get the database itself to connect. I am confident in the correct database name, username passowrd etc but am ...
Anton Gomes's user avatar
2 votes
2 answers
68 views

How to let site:example.com show results?

I set up a website example.com several months ago but it does not show up in site:example.com, so I submitted it in Google search console. After several days, I saw there were some pages indexed in ...
William's user avatar
  • 173
1 vote
0 answers
25 views

Sitemapindex containing nearly identical sitemaps? - is that a problem

I am currently in the process of migrating content from one domain to another domain, In this case I have some difficulty on how I should map the sitemap. I currently have a sitemapindex file, which ...
Carlton Banks's user avatar
1 vote
1 answer
53 views

How to place subdomain in the sitemap

A site made this sitemap for me: <urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http:/...
Saeed Asmani's user avatar
1 vote
1 answer
377 views

How soon does Bing start to obey a new robots.txt after accessing it?

I've recently noticed that Bing is crawling THOUSANDS of essentially duplicated pages on my site, and have updated my robots.txt as such: User-agent: * Crawl-delay: 1 Allow: / Disallow: /report-js-...
Codemonkey's user avatar
1 vote
1 answer
34 views

What do backslashes in WP generated robots.txt rules mean?

I am trying to implement a library to handle robots.txt and I am a bit confused about the meaning of some rules that I believe are generated by WordPress: Disallow: /?s=* Disallow: /*/?s=*\ Disallow: /...
alturkovic's user avatar
0 votes
1 answer
14 views

Is it possible to use a template for search spiders?

Is it possible to use a special template for search engines and another template for visitors? on the WordPress platform.
alyemenalghad's user avatar
0 votes
2 answers
293 views

How to Exclude Duck Duck Go Bot Traffic From Google analytics 4

I have been trying to create a filter for excluding Duckduck Go and Baidu search engine Bot Traffic from Google Analytics 4 for one of my websites. I am unable to create a filter for it. Internal &...
Olivia's user avatar
  • 61
1 vote
0 answers
214 views

Google Search Console detects issue with noindex pages excluded from sitemap, why? How to resolve?

Google emails me this: Message type: [WNC-10030322] Search Console Page indexing issues detected in [DOMAIN] To the owner of [DOMAIN]: Search Console has identified that your site is affected by 1 ...
2540625's user avatar
  • 141
1 vote
1 answer
71 views

How do password manager apps detect input tags?

How? Does it scrape the website for adjacent input tags, one with text and one with password? No matter which website I visit, it accurately prompts a dropdown (Touch ID on Safari) on the username ...
Ninad's user avatar
  • 11
1 vote
1 answer
176 views

Googlebot Spam / DDoS?

I recently saw a significant googlebot crawl on my website. We're talking 10-20 requests per second. It looked like googlebot, and seemed to be legit googlebot IPs, but the requests had query params ...
4m1r's user avatar
  • 111
2 votes
2 answers
306 views

Trying to block images from google image indexing but not working

I am trying to block images from google image search, but for some reason they are not working. I have a Wordpress self-hosted blog and I have full access to the backend. HTML of mydomain.com looks ...
cyanite's user avatar
  • 21
1 vote
0 answers
18 views

Website pages crawled entirely, thereafter only a 50% of pages are [duplicate]

Any technical or implementation tips on how to improve the crawling of site pages, when a given site used to have most pages crawled on its launch and thereafter only > 50% of them are. This ...
John Paul's user avatar
2 votes
1 answer
578 views

How to Resolve 'Blocked by robots.txt' Error Despite 'User-agent: * Allow: /' Directive?

I manage multiple custom-made websites hosted on the same server and with the same configuration, including sharing a Google Search Console account. Despite all sites utilizing the same robots.txt ...
Clipso's user avatar
  • 21
1 vote
1 answer
20 views

How can a react/vue apps like these rank just by using client side rendering?

By client-side rendering I mean, you can't see any content (h1,title,meta tags etc) in the source code, which is the only place where google bot can learn what a page is about. here is this site: ...
AbHi's user avatar
  • 11
2 votes
0 answers
25 views

How to manage internal links using short alternate URLs when the canonical URLs are complex to generate?

I am working on an existing e-commerce website where the canonical URL for product pages are long and complex to generate, hyphenated, and ending by the product id: https://example.com/brand-model-and-...
OuzoPower's user avatar
  • 357
1 vote
1 answer
238 views

How did Google crawler view my password-protected pages?

I have a site that is login-blocked/password-protected where Googlebot crawling has worked for 1+years and suddenly stopped. I am trying to figure out how this could have happened. It is a large site ...
James L.'s user avatar
  • 217
2 votes
1 answer
297 views

Can I set Cawl-delay in robots.txt to less than one second (like 0.1)?

I would like to start using the Crawl-delay on the robots.txt and saw that the value is in seconds. Is there a way to set it up below 1 second. Maybe 0.1? My goal is to allow crawlers not to run over ...
Milos Cuculovic's user avatar
2 votes
1 answer
4k views

What does User-agent: Mediapartners-Google mean?

I was going through the google support website and there they said: "To update your robots.txt file to grant our crawler access to your pages, remove the following two lines of text from your ...
Muhammad Ahmad's user avatar
0 votes
1 answer
454 views

Why website copy tools like Cyotek WebCopy and HTTrack cannot find files that search engines like Google can? [closed]

I would like to keep the target website private, but here are some details: It's a personal (as in single-author) public documentation / portfolio / blog sort of website It seems to be hosted using ...
Den's user avatar
  • 141
2 votes
1 answer
50 views

How to make a text website hostile only to machine-learning models while maintaining smooth experience for humans?

So, we're building a website that showcases some niche information and tries to present it in a way to build up a knowledge base. We want to maximize accessibility to people and spider bots, while ...
4amvim's user avatar
  • 23
2 votes
1 answer
81 views

Do article pages need to have an RSS Feed page?

I am working on a blog software. This blog has a home page, category pages and articles. I will add an RSS feed to this blog. I added the RSS feed to the home page by adding /feed to the end of the ...
ardacarofficial's user avatar
3 votes
1 answer
123 views

What is the actual HTML rendered to bots?

We have a product page in which 4 item carousels are present, showing mostly related products. All these 4 carousels are client side rendered. 1. Scenario1: Google rich results Now, we have observed ...
Ayyappa Gollu's user avatar
0 votes
1 answer
55 views

200 website code by Console [closed]

Can someone help me and let me know why google is not ranking quite some few of my sites from mine latest SEO project: https://kroatien-ferienwohnungen.at/ I see that plenty of pages are crawled but ...
Zirelco's user avatar
1 vote
1 answer
72 views

Is blocking indexing and crawling at the same time bad?

have a lot of pages that I want to disallow crawling and indexing on, because of crawl and index budgets. I can't just disable crawling because they are linked together with internal links and I can't ...
Supersporadic's user avatar
3 votes
2 answers
71 views

Prevent search engines indexing admin and other pages

My website has an Admin area which requires a login. There are also other pages that should not be indexed. I know that Robots.Txt entries or meta name="robots" content="noindex" ...
Clarendon VT Webmaster's user avatar
0 votes
1 answer
58 views

Allow and dissallow directory with # in robots.txt [duplicate]

I'd like to block and unblock following URL formats: Allow:https://example.com/buyer/#/search/catalog/Lubricating-Oils/Automotive-Lubricants Dissallow:https://example.com/buyer/#/search/product-...
suhanshukla's user avatar
5 votes
2 answers
1k views

How are 'scraped content' websites like diningandcooking.com able to rank so well despite having no original content?

I've stumbled upon a website called https://diningandcooking.com after finding out it was linking to one of my websites. When i went to the page that had the link on it i found out that its content ...
Maurice's user avatar
  • 397
2 votes
1 answer
51 views

Is disallowing all useragents sufficient to keep a site from being crawled by search engines?

We have a non-production version of our site that we want to prevent from being crawled. At some point in the past a non-production site was crawled and we still have search results pointing to it. I ...
Ian Pringle's user avatar
2 votes
0 answers
42 views

One day after putting my site online and registering it with Google I got spam mail. How?

I put my website online yesterday and registered it with Google at the same time. Before that, I didn't even put a single link to the site anywhere online. And I don't have any email address on my ...
Frank Conijn - Support Ukraine's user avatar
0 votes
0 answers
272 views

Google Search Console diplaying my URLs are indexed but it is not showing in search result by parameter site: search [duplicate]

Can anyone provide the solution about this issue:
Tops and Bottoms USA's user avatar
1 vote
1 answer
127 views

Can I Improve my website's Google rankings with a bot that searches for my site and clicks on it?

I have started a new website that I want like to obtain worldwide traffic for. And ofcourse, a key element is how to improve my Google search engine rank. I am considering to develop a bot that would ...
The Oracle's user avatar
2 votes
0 answers
30 views

Googlebot increased the number of requests for a certain URL even though it has no links

As part of regular Log File Analysis, I noticed in a URL that Google_bot was crawling like crazy. This URL is an Iframe that embed in a page. My thought was maybe this page gets a new link from ...
Lalikman's user avatar
3 votes
2 answers
936 views

How to configure robots.txt to disallow any URL/file/directory with a certain word in it?

I've seen basic articles online about allowing/disallowing certain subdirectories in robots.txt, but I have a more detailed need than that. Google search console keeps trying to discover/crawl certain ...
ravl13's user avatar
  • 45
2 votes
0 answers
336 views

What is correct way to stop query indexing using Robots.txt?

How to stop indexing search result using Robots.txt? hello. Recently, I'm having an issue with indexed pages in Google Search Console growing 10x. These pages are mostly search query (/?s=) and ...
cheonmu's user avatar
  • 21
2 votes
2 answers
847 views

Will Googlebot click on tab navigation / buttons and index links?

I have the following UI: There are tabs which are <button>'s and when you select one there will be <a>-Links to different pages. It seems like Google will only index the Links on the ...
Lars Flieger's user avatar
4 votes
3 answers
1k views

Bing refuses to index my site at all

I have been trying for months now to get Bing to index my personal website, to no avail. I'm not talking about ranking, as I expect it to rank quite low. It is invisible. I don't care about Bing in ...
herrbischoff's user avatar

1
2 3 4 5
19