Questions tagged [web-crawlers]
A computer program that accesses web pages for various purposes (to scrape content, to provide search engines with information about your site, etc.)
903
questions
1
vote
0
answers
60
views
Should Feed/RSS Be Blocked from Robots.txt?
Should Feed/RSS be blocked via robots.txt?
Disallow: /*feed*
Disallow: /*rss*
If yes, is this the correct way to apply?
If you are sending Feed/RSS somewhere or if you have a subscription system, of ...
- 41
4
votes
1
answer
43
views
How to protect from spam when the legislation forces you to display the email address?
Context
I want to protect a ecommerce website from email spam, whilst the legislation in my country makes mandatory to publish an email contact address on ecommerce sites.
As an additional constraint, ...
- 337
3
votes
1
answer
121
views
SEMRush Site Audit can't crawl PDFs on my site, is this an issue with my server?
My agency uses SEMRush, and we especially find the Site Audit function helpful for spotting potential issues with our SEO and UX.
However of late, SEMRush has been flagging PDFs quite frequently, ...
- 2,209
1
vote
1
answer
30
views
What is the Query to add in robots.txt for blocking URLs has some special characters?
I was going through the Google Search Console pages issues and one of the issue reason is "Alternate page with proper canonical tag" and this has some URLs which look like below
example.com/...
0
votes
1
answer
142
views
Use passive listeners to improve scrolling performance
Google's Lighthouse tests for CWV ran on my website include an issue stating that touch and wheel event listeners are not marked as passive.
How and where to fix this?
I have tried to check my file ...
1
vote
0
answers
75
views
Connecting to MySQL database hosted on CPanel
I have followed all of the usual steps in setting up a pipeline but cannot seem to get the database itself to connect. I am confident in the correct database name, username passowrd etc but am ...
- 11
2
votes
2
answers
62
views
How to let site:example.com show results?
I set up a website example.com several months ago but it does not show up in site:example.com, so I submitted it in Google search console. After several days, I saw there were some pages indexed in ...
- 173
1
vote
0
answers
25
views
Sitemapindex containing nearly identical sitemaps? - is that a problem
I am currently in the process of migrating content from one domain to another domain,
In this case I have some difficulty on how I should map the sitemap.
I currently have a sitemapindex file, which ...
- 111
1
vote
1
answer
45
views
How to place subdomain in the sitemap
A site made this sitemap for me:
<urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http:/...
- 11
1
vote
1
answer
110
views
How soon does Bing start to obey a new robots.txt after accessing it?
I've recently noticed that Bing is crawling THOUSANDS of essentially duplicated pages on my site, and have updated my robots.txt as such:
User-agent: *
Crawl-delay: 1
Allow: /
Disallow: /report-js-...
- 399
1
vote
1
answer
25
views
What do backslashes in WP generated robots.txt rules mean?
I am trying to implement a library to handle robots.txt and I am a bit confused about the meaning of some rules that I believe are generated by WordPress:
Disallow: /?s=*
Disallow: /*/?s=*\
Disallow: /...
- 113
0
votes
1
answer
14
views
Is it possible to use a template for search spiders?
Is it possible to use a special template for search engines and another template for visitors? on the WordPress platform.
0
votes
0
answers
111
views
"GET / HTTP/1.1" requests getting 404 from malicious agents and bots?
I was checking my server logs and I see some requests simply for "GET / HTTP/1.1" are getting 404 error but only from seemingly malicious users and bots. I thought 404 was only server-side ...
- 143
0
votes
2
answers
204
views
How to Exclude Duck Duck Go Bot Traffic From Google analytics 4
I have been trying to create a filter for excluding Duckduck Go and Baidu search engine Bot Traffic from Google Analytics 4 for one of my websites. I am unable to create a filter for it. Internal &...
- 61
1
vote
0
answers
155
views
Google Search Console detects issue with noindex pages excluded from sitemap, why? How to resolve?
Google emails me this:
Message type: [WNC-10030322]
Search Console
Page indexing issues detected in [DOMAIN]
To the owner of [DOMAIN]:
Search Console has identified that your site is affected by 1 ...
- 111
0
votes
0
answers
20
views
Reproduce the API links appearing in Google search results
We have the following problem. The links to our API (that doesn't require any authentication) have suddenly started showing up in Google and Bing search results. As far as we know, these links have ...
- 1
1
vote
1
answer
51
views
How do password manager apps detect input tags?
How? Does it scrape the website for adjacent input tags, one with text and one with password? No matter which website I visit, it accurately prompts a dropdown (Touch ID on Safari) on the username ...
- 11
1
vote
1
answer
85
views
Googlebot Spam / DDoS?
I recently saw a significant googlebot crawl on my website. We're talking 10-20 requests per second. It looked like googlebot, and seemed to be legit googlebot IPs, but the requests had query params ...
- 111
2
votes
2
answers
157
views
Trying to block images from google image indexing but not working
I am trying to block images from google image search, but for some reason they are not working. I have a Wordpress self-hosted blog and I have full access to the backend. HTML of mydomain.com looks ...
- 21
0
votes
0
answers
37
views
Why does our site name keep reverting back to an incorrect name?
We have two websites living under two different domains. Domain A and domain B. Domain B was decommissioned in February and migrated over to domain A with a URL path /en-us/. All of the old URLs from ...
1
vote
0
answers
17
views
Website pages crawled entirely, thereafter only a 50% of pages are [duplicate]
Any technical or implementation tips on how to improve the crawling of site pages, when a given site used to have most pages crawled on its launch and thereafter only > 50% of them are. This ...
- 23
2
votes
1
answer
231
views
How to Resolve 'Blocked by robots.txt' Error Despite 'User-agent: * Allow: /' Directive?
I manage multiple custom-made websites hosted on the same server and with the same configuration, including sharing a Google Search Console account. Despite all sites utilizing the same robots.txt ...
- 21
1
vote
1
answer
18
views
How can a react/vue apps like these rank just by using client side rendering?
By client-side rendering I mean, you can't see any content (h1,title,meta tags etc) in the source code, which is the only place where google bot can learn what a page is about.
here is this site: ...
- 11
2
votes
0
answers
23
views
How to manage internal links using short alternate URLs when the canonical URLs are complex to generate?
I am working on an existing e-commerce website where the canonical URL for product pages are long and complex to generate, hyphenated, and ending by the product id: https://example.com/brand-model-and-...
- 337
1
vote
1
answer
122
views
How did Google crawler view my password-protected pages?
I have a site that is login-blocked/password-protected where Googlebot crawling has worked for 1+years and suddenly stopped.
I am trying to figure out how this could have happened. It is a large site ...
- 217
2
votes
1
answer
167
views
Can I set Cawl-delay in robots.txt to less than one second (like 0.1)?
I would like to start using the Crawl-delay on the robots.txt and saw that the value is in seconds. Is there a way to set it up below 1 second. Maybe 0.1?
My goal is to allow crawlers not to run over ...
- 121
0
votes
0
answers
108
views
Adding Custom robots.txt in blogger
I was looking at examples to add the file to my blogger website and came across a code:
User-agent: Mediapartners-Google
Disallow:
User-agent: *
Disallow: /search
Allow: /
Sitemap:https://example.com/...
2
votes
1
answer
2k
views
What does User-agent: Mediapartners-Google mean?
I was going through the google support website and there they said:
"To update your robots.txt file to grant our crawler access to your pages, remove the following two lines of text from your ...
0
votes
1
answer
291
views
Why website copy tools like Cyotek WebCopy and HTTrack cannot find files that search engines like Google can? [closed]
I would like to keep the target website private, but here are some details:
It's a personal (as in single-author) public documentation / portfolio / blog sort of website
It seems to be hosted using ...
- 141
0
votes
0
answers
58
views
Alternate Canonical Tags error on Google Search Console
I recently started my blogging site and after a few days on Google Search Console I got an error saying alternate page with proper canonical tag. Now I searched the internet and found the fix. A line ...
2
votes
1
answer
45
views
How to make a text website hostile only to machine-learning models while maintaining smooth experience for humans?
So, we're building a website that showcases some niche information and tries to present it in a way to build up a knowledge base. We want to maximize accessibility to people and spider bots, while ...
- 23
2
votes
1
answer
73
views
Do article pages need to have an RSS Feed page?
I am working on a blog software. This blog has a home page, category pages and articles.
I will add an RSS feed to this blog. I added the RSS feed to the home page by adding /feed to the end of the ...
- 1,017
3
votes
1
answer
109
views
What is the actual HTML rendered to bots?
We have a product page in which 4 item carousels are present, showing mostly related products. All these 4 carousels are client side rendered.
1. Scenario1: Google rich results
Now, we have observed ...
- 141
0
votes
1
answer
34
views
200 website code by Console [closed]
Can someone help me and let me know why google is not ranking quite some few of my sites from mine latest SEO project:
https://kroatien-ferienwohnungen.at/
I see that plenty of pages are crawled but ...
- 3
1
vote
1
answer
52
views
Is blocking indexing and crawling at the same time bad?
have a lot of pages that I want to disallow crawling and indexing on, because of crawl and index budgets. I can't just disable crawling because they are linked together with internal links and I can't ...
- 65
3
votes
2
answers
41
views
Prevent search engines indexing admin and other pages
My website has an Admin area which requires a login. There are also other pages that should not be indexed. I know that Robots.Txt entries or meta name="robots" content="noindex" ...
0
votes
1
answer
45
views
Allow and dissallow directory with # in robots.txt [duplicate]
I'd like to block and unblock following URL formats:
Allow:https://example.com/buyer/#/search/catalog/Lubricating-Oils/Automotive-Lubricants
Dissallow:https://example.com/buyer/#/search/product-...
5
votes
2
answers
1k
views
How are 'scraped content' websites like diningandcooking.com able to rank so well despite having no original content?
I've stumbled upon a website called https://diningandcooking.com after finding out it was linking to one of my websites. When i went to the page that had the link on it i found out that its content ...
- 397
2
votes
1
answer
47
views
Is disallowing all useragents sufficient to keep a site from being crawled by search engines?
We have a non-production version of our site that we want to prevent from being crawled. At some point in the past a non-production site was crawled and we still have search results pointing to it. I ...
- 123
2
votes
0
answers
41
views
One day after putting my site online and registering it with Google I got spam mail. How?
I put my website online yesterday and registered it with Google at the same time. Before that, I didn't even put a single link to the site anywhere online. And I don't have any email address on my ...
0
votes
0
answers
229
views
Google Search Console diplaying my URLs are indexed but it is not showing in search result by parameter site: search [duplicate]
Can anyone provide the solution about this issue:
1
vote
1
answer
93
views
Can I Improve my website's Google rankings with a bot that searches for my site and clicks on it?
I have started a new website that I want like to obtain worldwide traffic for. And ofcourse, a key element is how to improve my Google search engine rank.
I am considering to develop a bot that would ...
- 143
2
votes
0
answers
29
views
Googlebot increased the number of requests for a certain URL even though it has no links
As part of regular Log File Analysis, I noticed in a URL that Google_bot was crawling like crazy. This URL is an Iframe that embed in a page. My thought was maybe this page gets a new link from ...
- 51
3
votes
2
answers
535
views
How to configure robots.txt to disallow any URL/file/directory with a certain word in it?
I've seen basic articles online about allowing/disallowing certain subdirectories in robots.txt, but I have a more detailed need than that.
Google search console keeps trying to discover/crawl certain ...
- 45
2
votes
0
answers
234
views
What is correct way to stop query indexing using Robots.txt?
How to stop indexing search result using Robots.txt?
hello.
Recently, I'm having an issue with indexed pages in Google Search Console growing 10x.
These pages are mostly search query (/?s=) and ...
- 21
2
votes
2
answers
549
views
Will Googlebot click on tab navigation / buttons and index links?
I have the following UI:
There are tabs which are <button>'s and when you select one there will be <a>-Links to different pages. It seems like Google will only index the Links on the ...
- 153
4
votes
3
answers
1k
views
Bing refuses to index my site at all
I have been trying for months now to get Bing to index my personal website, to no avail. I'm not talking about ranking, as I expect it to rank quite low. It is invisible. I don't care about Bing in ...
- 175
1
vote
1
answer
128
views
Will Google crawl links with other nested HTML elements?
I have a link like this
<a href="/some/relative/link">
<div class="container">
Some Content
</div>
</a>
On the Google Search Console, it says that ...
- 113
-1
votes
1
answer
554
views
Does googlebot crawl the variable name of a php [closed]
Does Googlebot crawl the dynamic variable name aside from its value? For example in my PHP I use the variable _company_name. In the front-end is is _company_name saved in bot crawl also?
- 27
0
votes
0
answers
119
views
Bingbot issues and crawling my website root domain [duplicate]
I seem to have this issue with a few different website's but bing is unable to crawl (or fetch the pages).
I have done a cURL via windows and it connected properly however every URL live inspection ...
