All Questions
Tagged with crawling or web-crawlers
913 questions
1
vote
2
answers
25
views
Domain change, 301 Redirects, Site Map & Indexing
I changed my old domain name to a shorter domain name. I set up 301 redirects and uploaded the XML site map of the new website to Google Search Console. Google indexed the new domain, but I still see ...
- 11
0
votes
1
answer
37
views
Why does Google not like my structured data
This is for the web page https://louishowe-dev.azurewebsites.net/Organization/Profile?id=30015&crawler=true
Google's Schema Markup Validator is able to read all the data. (so good)
Google's Rich ...
- 139
0
votes
0
answers
12
views
Testing sitemap, rich results, etc. on my dev slot
How can I set things up so that I can access my dev slot with the Google developer console, but in a way that Google will not index the page.
I can't use robots.txt because we push our app to ...
- 139
2
votes
1
answer
58
views
Is there a crawl-delay limit?
Does anyone know if there is a max limit to Crawl-Delay time? I read people set to 10 to keep bots down to 8,600 crawls a day, but I would like to see the limit even more.
I set mine to 30, which ...
0
votes
0
answers
42
views
Do two word bots need escaping with %20 or are spaces Ok in robots.txt
I want to tell Baidu not to crawl. One of their crawlers is Baidu Spider.
Should that be:
User-agent: Baidu%20Spider
or
User-agent: Baidu Spider
- 47
0
votes
0
answers
25
views
Issues with Googlebot crawling and negative number pages
I am dealing with very heavy load from Googlebot, I have hundreds of sites running on multiple IPs all on the same server and googlebot is constantly crawling these sites causing a lot of availability ...
- 101
0
votes
1
answer
35
views
Do indexer bots set navigator.webdriver property to true?
The webdriver read-only property of the navigator interface indicates whether the user agent is controlled by automation.
Source
I couldn't find anywhere if it includes indexing crawlers such as ...
- 175
1
vote
1
answer
70
views
Do web crawlers clear their browser cache?
Suppose you make a website and you forget to set the proper cache control headers, and your index.html is set to be cached and never expire. You fix this, and after clearing your browser cache this ...
- 113
1
vote
2
answers
3k
views
How do I stop Amazon from crawling a website?
I would like to prevent Amazon from scraping product data on my website. So I found this document: https://developer.amazon.com/amazonbot
And this example:
User-agent: Amazonbot # Amazon's ...
Rhothgar
0
votes
0
answers
12
views
Why aren't search engines showing my site anymore? [duplicate]
I have created a site as a home project but for some reason, it stopped showing my site in Google. I connected my site to Google Search Console and most of the pages are indexed. At least I can see it ...
- 1
0
votes
0
answers
20
views
Weird Apache requests
I've recently migrated a client from a WIX site to a Wordpress site (running on an Apache server)
I am seeing quite a few requests in the apache log file in the format
site.name:443 2.2.2.2 - - [19/...
- 8,296
3
votes
1
answer
542
views
Google Search Console: My "indexed" and "not-indexed" pages have flipped. I now have more "not-indexed" pages
Web App Google Search Console Index: Crossover
I have a web app with a lot of dynamically generated pages.
Each page has similar html layout with different text/image content, as well as a unique url.
...
- 33
0
votes
0
answers
21
views
Exceed crawl limits for a website [duplicate]
In previous month i created a lot of pages (> 200K), and google indexed it. After some time a relized that i don't need some of these pages and created a new ones, but google don't indexing it. I ...
- 23
2
votes
1
answer
68
views
JSON Files getting crawled suddenly
Just noticed that GSC has recently taken an interest in the .json files that are associated to all of the product pages on Shopify, and is now crawling them, but wasn't before.
Should I be worried ...
- 21
0
votes
0
answers
32
views
How does Serpi protect from web crawling restrictions
I'm thinking of making a website to find the lowest price of an item on the net. The problem is that in the terms of use of Google search API it says that it is forbidden to:
Scrape, build databases, ...
1
vote
0
answers
560
views
Should Feed/RSS Be Blocked from Robots.txt?
Should Feed/RSS be blocked via robots.txt?
Disallow: /*feed*
Disallow: /*rss*
If yes, is this the correct way to apply?
If you are sending Feed/RSS somewhere or if you have a subscription system, of ...
- 41
4
votes
1
answer
60
views
How to protect from spam when the legislation forces you to display the email address?
Context
I want to protect a ecommerce website from email spam, whilst the legislation in my country makes mandatory to publish an email contact address on ecommerce sites.
As an additional constraint, ...
- 357
3
votes
1
answer
341
views
SEMRush Site Audit can't crawl PDFs on my site, is this an issue with my server?
My agency uses SEMRush, and we especially find the Site Audit function helpful for spotting potential issues with our SEO and UX.
However of late, SEMRush has been flagging PDFs quite frequently, ...
- 2,297
1
vote
1
answer
49
views
What is the Query to add in robots.txt for blocking URLs has some special characters?
I was going through the Google Search Console pages issues and one of the issue reason is "Alternate page with proper canonical tag" and this has some URLs which look like below
example.com/...
0
votes
1
answer
274
views
Use passive listeners to improve scrolling performance
Google's Lighthouse tests for CWV ran on my website include an issue stating that touch and wheel event listeners are not marked as passive.
How and where to fix this?
I have tried to check my file ...
1
vote
0
answers
183
views
Connecting to MySQL database hosted on CPanel
I have followed all of the usual steps in setting up a pipeline but cannot seem to get the database itself to connect. I am confident in the correct database name, username passowrd etc but am ...
- 11
2
votes
2
answers
71
views
How to let site:example.com show results?
I set up a website example.com several months ago but it does not show up in site:example.com, so I submitted it in Google search console. After several days, I saw there were some pages indexed in ...
- 173
1
vote
0
answers
26
views
Sitemapindex containing nearly identical sitemaps? - is that a problem
I am currently in the process of migrating content from one domain to another domain,
In this case I have some difficulty on how I should map the sitemap.
I currently have a sitemapindex file, which ...
- 111
1
vote
1
answer
57
views
How to place subdomain in the sitemap
A site made this sitemap for me:
<urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http:/...
- 11
1
vote
1
answer
696
views
How soon does Bing start to obey a new robots.txt after accessing it?
I've recently noticed that Bing is crawling THOUSANDS of essentially duplicated pages on my site, and have updated my robots.txt as such:
User-agent: *
Crawl-delay: 1
Allow: /
Disallow: /report-js-...
- 419
1
vote
1
answer
48
views
What do backslashes in WP generated robots.txt rules mean?
I am trying to implement a library to handle robots.txt and I am a bit confused about the meaning of some rules that I believe are generated by WordPress:
Disallow: /?s=*
Disallow: /*/?s=*\
Disallow: /...
- 113
0
votes
1
answer
15
views
Is it possible to use a template for search spiders?
Is it possible to use a special template for search engines and another template for visitors? on the WordPress platform.
0
votes
2
answers
355
views
How to Exclude Duck Duck Go Bot Traffic From Google analytics 4
I have been trying to create a filter for excluding Duckduck Go and Baidu search engine Bot Traffic from Google Analytics 4 for one of my websites. I am unable to create a filter for it. Internal &...
- 61
1
vote
0
answers
290
views
Google Search Console detects issue with noindex pages excluded from sitemap, why? How to resolve?
Google emails me this:
Message type: [WNC-10030322]
Search Console
Page indexing issues detected in [DOMAIN]
To the owner of [DOMAIN]:
Search Console has identified that your site is affected by 1 ...
- 113
1
vote
1
answer
84
views
How do password manager apps detect input tags?
How? Does it scrape the website for adjacent input tags, one with text and one with password? No matter which website I visit, it accurately prompts a dropdown (Touch ID on Safari) on the username ...
- 11
1
vote
1
answer
281
views
Googlebot Spam / DDoS?
I recently saw a significant googlebot crawl on my website. We're talking 10-20 requests per second. It looked like googlebot, and seemed to be legit googlebot IPs, but the requests had query params ...
- 111
2
votes
2
answers
438
views
Trying to block images from google image indexing but not working
I am trying to block images from google image search, but for some reason they are not working. I have a Wordpress self-hosted blog and I have full access to the backend. HTML of mydomain.com looks ...
- 21
1
vote
0
answers
18
views
Website pages crawled entirely, thereafter only a 50% of pages are [duplicate]
Any technical or implementation tips on how to improve the crawling of site pages, when a given site used to have most pages crawled on its launch and thereafter only > 50% of them are. This ...
- 23
2
votes
1
answer
966
views
How to Resolve 'Blocked by robots.txt' Error Despite 'User-agent: * Allow: /' Directive?
I manage multiple custom-made websites hosted on the same server and with the same configuration, including sharing a Google Search Console account. Despite all sites utilizing the same robots.txt ...
- 21
1
vote
1
answer
23
views
How can a react/vue apps like these rank just by using client side rendering?
By client-side rendering I mean, you can't see any content (h1,title,meta tags etc) in the source code, which is the only place where google bot can learn what a page is about.
here is this site: ...
- 11
2
votes
0
answers
26
views
How to manage internal links using short alternate URLs when the canonical URLs are complex to generate?
I am working on an existing e-commerce website where the canonical URL for product pages are long and complex to generate, hyphenated, and ending by the product id: https://example.com/brand-model-and-...
- 357
1
vote
1
answer
349
views
How did Google crawler view my password-protected pages?
I have a site that is login-blocked/password-protected where Googlebot crawling has worked for 1+years and suddenly stopped.
I am trying to figure out how this could have happened. It is a large site ...
- 217
2
votes
1
answer
397
views
Can I set Cawl-delay in robots.txt to less than one second (like 0.1)?
I would like to start using the Crawl-delay on the robots.txt and saw that the value is in seconds. Is there a way to set it up below 1 second. Maybe 0.1?
My goal is to allow crawlers not to run over ...
- 123
3
votes
2
answers
5k
views
What does User-agent: Mediapartners-Google mean?
I was going through the google support website and there they said:
"To update your robots.txt file to grant our crawler access to your pages, remove the following two lines of text from your ...
- 102
0
votes
1
answer
609
views
Why website copy tools like Cyotek WebCopy and HTTrack cannot find files that search engines like Google can? [closed]
I would like to keep the target website private, but here are some details:
It's a personal (as in single-author) public documentation / portfolio / blog sort of website
It seems to be hosted using ...
- 141
2
votes
1
answer
55
views
How to make a text website hostile only to machine-learning models while maintaining smooth experience for humans?
So, we're building a website that showcases some niche information and tries to present it in a way to build up a knowledge base. We want to maximize accessibility to people and spider bots, while ...
- 23
2
votes
1
answer
90
views
Do article pages need to have an RSS Feed page?
I am working on a blog software. This blog has a home page, category pages and articles.
I will add an RSS feed to this blog. I added the RSS feed to the home page by adding /feed to the end of the ...
- 1,055
3
votes
1
answer
146
views
What is the actual HTML rendered to bots?
We have a product page in which 4 item carousels are present, showing mostly related products. All these 4 carousels are client side rendered.
1. Scenario1: Google rich results
Now, we have observed ...
- 141
0
votes
1
answer
56
views
200 website code by Console [closed]
Can someone help me and let me know why google is not ranking quite some few of my sites from mine latest SEO project:
https://kroatien-ferienwohnungen.at/
I see that plenty of pages are crawled but ...
- 3
1
vote
1
answer
94
views
Is blocking indexing and crawling at the same time bad?
have a lot of pages that I want to disallow crawling and indexing on, because of crawl and index budgets. I can't just disable crawling because they are linked together with internal links and I can't ...
3
votes
2
answers
99
views
Prevent search engines indexing admin and other pages
My website has an Admin area which requires a login. There are also other pages that should not be indexed. I know that Robots.Txt entries or meta name="robots" content="noindex" ...
0
votes
1
answer
67
views
Allow and dissallow directory with # in robots.txt [duplicate]
I'd like to block and unblock following URL formats:
Allow:https://example.com/buyer/#/search/catalog/Lubricating-Oils/Automotive-Lubricants
Dissallow:https://example.com/buyer/#/search/product-...
4
votes
2
answers
1k
views
How are 'scraped content' websites like diningandcooking.com able to rank so well despite having no original content?
I've stumbled upon a website called https://diningandcooking.com after finding out it was linking to one of my websites. When i went to the page that had the link on it i found out that its content ...
- 387
1
vote
1
answer
59
views
Is disallowing all useragents sufficient to keep a site from being crawled by search engines?
We have a non-production version of our site that we want to prevent from being crawled. At some point in the past a non-production site was crawled and we still have search results pointing to it. I ...
- 113
2
votes
0
answers
43
views
One day after putting my site online and registering it with Google I got spam mail. How?
I put my website online yesterday and registered it with Google at the same time. Before that, I didn't even put a single link to the site anywhere online. And I don't have any email address on my ...
