All Questions
Tagged with crawling or web-crawlers
913 questions
1
vote
2
answers
25
views
Domain change, 301 Redirects, Site Map & Indexing
I changed my old domain name to a shorter domain name. I set up 301 redirects and uploaded the XML site map of the new website to Google Search Console. Google indexed the new domain, but I still see ...
- 11
1
vote
2
answers
3k
views
How do I stop Amazon from crawling a website?
I would like to prevent Amazon from scraping product data on my website. So I found this document: https://developer.amazon.com/amazonbot
And this example:
User-agent: Amazonbot # Amazon's ...
Rhothgar
0
votes
0
answers
42
views
Do two word bots need escaping with %20 or are spaces Ok in robots.txt
I want to tell Baidu not to crawl. One of their crawlers is Baidu Spider.
Should that be:
User-agent: Baidu%20Spider
or
User-agent: Baidu Spider
- 47
0
votes
1
answer
37
views
Why does Google not like my structured data
This is for the web page https://louishowe-dev.azurewebsites.net/Organization/Profile?id=30015&crawler=true
Google's Schema Markup Validator is able to read all the data. (so good)
Google's Rich ...
- 139
2
votes
1
answer
59
views
Is there a crawl-delay limit?
Does anyone know if there is a max limit to Crawl-Delay time? I read people set to 10 to keep bots down to 8,600 crawls a day, but I would like to see the limit even more.
I set mine to 30, which ...
3
votes
1
answer
542
views
Google Search Console: My "indexed" and "not-indexed" pages have flipped. I now have more "not-indexed" pages
Web App Google Search Console Index: Crossover
I have a web app with a lot of dynamically generated pages.
Each page has similar html layout with different text/image content, as well as a unique url.
...
- 33
3
votes
2
answers
5k
views
What does User-agent: Mediapartners-Google mean?
I was going through the google support website and there they said:
"To update your robots.txt file to grant our crawler access to your pages, remove the following two lines of text from your ...
- 102
1
vote
1
answer
70
views
Do web crawlers clear their browser cache?
Suppose you make a website and you forget to set the proper cache control headers, and your index.html is set to be cached and never expire. You fix this, and after clearing your browser cache this ...
- 113
14
votes
8
answers
6k
views
Is it possible to keep the existence of a domain secret?
If I bought a domain like a9j47fn83jd8j49.tld, and only a friend and I ever visit it, who would know about it? It sounds like for most TLDs, there is some reporting that has to happen after you buy a ...
- 243
0
votes
1
answer
35
views
Do indexer bots set navigator.webdriver property to true?
The webdriver read-only property of the navigator interface indicates whether the user agent is controlled by automation.
Source
I couldn't find anywhere if it includes indexing crawlers such as ...
- 175
4
votes
1
answer
60
views
How to protect from spam when the legislation forces you to display the email address?
Context
I want to protect a ecommerce website from email spam, whilst the legislation in my country makes mandatory to publish an email contact address on ecommerce sites.
As an additional constraint, ...
- 357
3
votes
1
answer
341
views
SEMRush Site Audit can't crawl PDFs on my site, is this an issue with my server?
My agency uses SEMRush, and we especially find the Site Audit function helpful for spotting potential issues with our SEO and UX.
However of late, SEMRush has been flagging PDFs quite frequently, ...
- 2,297
2
votes
1
answer
68
views
JSON Files getting crawled suddenly
Just noticed that GSC has recently taken an interest in the .json files that are associated to all of the product pages on Shopify, and is now crawling them, but wasn't before.
Should I be worried ...
- 21
11
votes
4
answers
6k
views
Prevent Google from thinking my own bot is faking traffic on my website
I have a large website which I want to be visited by my own bot early on the morning, every day (for caching purpose). I want my bot visits 10,000 pages per day.
My natural traffic is about 5000 pages ...
- 111
1
vote
1
answer
84
views
How do password manager apps detect input tags?
How? Does it scrape the website for adjacent input tags, one with text and one with password? No matter which website I visit, it accurately prompts a dropdown (Touch ID on Safari) on the username ...
- 11
0
votes
0
answers
20
views
Weird Apache requests
I've recently migrated a client from a WIX site to a Wordpress site (running on an Apache server)
I am seeing quite a few requests in the apache log file in the format
site.name:443 2.2.2.2 - - [19/...
- 8,296
0
votes
0
answers
12
views
Why aren't search engines showing my site anymore? [duplicate]
I have created a site as a home project but for some reason, it stopped showing my site in Google. I connected my site to Google Search Console and most of the pages are indexed. At least I can see it ...
- 1
0
votes
2
answers
355
views
How to Exclude Duck Duck Go Bot Traffic From Google analytics 4
I have been trying to create a filter for excluding Duckduck Go and Baidu search engine Bot Traffic from Google Analytics 4 for one of my websites. I am unable to create a filter for it. Internal &...
- 61
4
votes
3
answers
2k
views
Bing refuses to index my site at all
I have been trying for months now to get Bing to index my personal website, to no avail. I'm not talking about ranking, as I expect it to rank quite low. It is invisible. I don't care about Bing in ...
- 175
0
votes
1
answer
274
views
Use passive listeners to improve scrolling performance
Google's Lighthouse tests for CWV ran on my website include an issue stating that touch and wheel event listeners are not marked as passive.
How and where to fix this?
I have tried to check my file ...
2
votes
1
answer
628
views
Will redirecting to a mailto URL prevent spam bots from harvesting the email address?
On https://www.heartinternet.uk/blog/15-ways-to-hide-your-email-address/, under the sub heading Replacing with a PHP script, a method to hide email addresses is presented that I like very much. On the ...
4
votes
2
answers
1k
views
How are 'scraped content' websites like diningandcooking.com able to rank so well despite having no original content?
I've stumbled upon a website called https://diningandcooking.com after finding out it was linking to one of my websites. When i went to the page that had the link on it i found out that its content ...
- 387
1
vote
0
answers
560
views
Should Feed/RSS Be Blocked from Robots.txt?
Should Feed/RSS be blocked via robots.txt?
Disallow: /*feed*
Disallow: /*rss*
If yes, is this the correct way to apply?
If you are sending Feed/RSS somewhere or if you have a subscription system, of ...
- 41
2
votes
2
answers
1k
views
Should I exclude the assets folder from search engine crawlers in angular
I'm working with angular and had added the assets folder as element to exclude from the search engine crawlers, into my robots.txt file.
But I now get the following error while testing the site health ...
- 163
0
votes
0
answers
32
views
How does Serpi protect from web crawling restrictions
I'm thinking of making a website to find the lowest price of an item on the net. The problem is that in the terms of use of Google search API it says that it is forbidden to:
Scrape, build databases, ...
2
votes
1
answer
966
views
How to Resolve 'Blocked by robots.txt' Error Despite 'User-agent: * Allow: /' Directive?
I manage multiple custom-made websites hosted on the same server and with the same configuration, including sharing a Google Search Console account. Despite all sites utilizing the same robots.txt ...
- 21
0
votes
0
answers
21
views
Exceed crawl limits for a website [duplicate]
In previous month i created a lot of pages (> 200K), and google indexed it. After some time a relized that i don't need some of these pages and created a new ones, but google don't indexing it. I ...
- 23
1
vote
0
answers
183
views
Connecting to MySQL database hosted on CPanel
I have followed all of the usual steps in setting up a pipeline but cannot seem to get the database itself to connect. I am confident in the correct database name, username passowrd etc but am ...
- 11
2
votes
2
answers
71
views
How to let site:example.com show results?
I set up a website example.com several months ago but it does not show up in site:example.com, so I submitted it in Google search console. After several days, I saw there were some pages indexed in ...
- 173
1
vote
1
answer
349
views
How did Google crawler view my password-protected pages?
I have a site that is login-blocked/password-protected where Googlebot crawling has worked for 1+years and suddenly stopped.
I am trying to figure out how this could have happened. It is a large site ...
- 217
2
votes
1
answer
397
views
Can I set Cawl-delay in robots.txt to less than one second (like 0.1)?
I would like to start using the Crawl-delay on the robots.txt and saw that the value is in seconds. Is there a way to set it up below 1 second. Maybe 0.1?
My goal is to allow crawlers not to run over ...
- 123
3
votes
1
answer
443
views
Why Googlebot is crawling only one page?
I am trying to understand why Google bot is crawling only one page only on my website.
At the Google console I registered a small XML sitemap with 9 URLs. Only one URL I marked as a high priority (0.8)...
- 227
18
votes
3
answers
4k
views
Should I publish an empty ads.txt file for a site that does not run any ads?
I have seen a few 404 errors when ads.txt is requested on some of the sites that I manage. I Guess the answer may be identical to the question about an empty robots.txt file or no file at all (just ...
- 582
4
votes
1
answer
2k
views
Does Googlebot crawl JS or JSON inside an HTML page?
My Google Search tool is complaining about certain URLs it found on my site. These are intentionally unusable URLs; I'm not using them in any links. Think of them as unused meta-data.
A) In one ...
- 143
2
votes
1
answer
1k
views
Google Indexing Tons of Non-Existent URLs
A client that I am working for has come to me with a problem. Their Google Adwords account has been disabled because there is a bunch of traffic to crazy non-existent pages on their website. Here are ...
- 23
6
votes
4
answers
4k
views
How I can ensure that a link sent via email is opened only via user clicks from a mail client and not by bots?
In my web application I generate links in the following format:
https://example.com/^token^
Each link according to my specs is sent via email and should not allow robots to scrape it. In order to ...
15
votes
7
answers
26k
views
How to properly (dis)allow the archive.org bot? Did things change, if so when?
I have a website that I mostly don't want to be indexed by search engines, but I do want to preserve it for eternity on archive.org. So my robots.txt starts with this:
User-agent: *
Disallow: /
...
- 1,600
2
votes
2
answers
1k
views
Will Googlebot click on tab navigation / buttons and index links?
I have the following UI:
There are tabs which are <button>'s and when you select one there will be <a>-Links to different pages. It seems like Google will only index the Links on the ...
- 153
1
vote
2
answers
2k
views
Too many hits from GoogleBot
The last week I have about 500.000 hits/day from googlebot in one of my websites. As a result, the server has serious issues to handle the rest of the requests.
What I have done
Verified that it is ...
- 193
4
votes
2
answers
340
views
Do SEO analysis bots have any advantage in terms of website traffic?
I have loads of traffic from SEO analysis bots (which are crawling for link analysis purpose only, rather than indexing for search engines) such as
AhrefsBot/7.0; +http://ahrefs.com/robot/
SemrushBot/...
- 452
-1
votes
1
answer
583
views
Does googlebot crawl the variable name of a php [closed]
Does Googlebot crawl the dynamic variable name aside from its value? For example in my PHP I use the variable _company_name. In the front-end is is _company_name saved in bot crawl also?
- 27
1
vote
1
answer
445
views
How does authentication-required content get indexed/crawled?
Assume we have a VOD platform to watch online videos and it requires authentication to show that video or any kind of content. User has to be signed up and after that, they can access the videos etc. ...
- 111
3
votes
1
answer
146
views
What is the actual HTML rendered to bots?
We have a product page in which 4 item carousels are present, showing mostly related products. All these 4 carousels are client side rendered.
1. Scenario1: Google rich results
Now, we have observed ...
- 141
2
votes
2
answers
438
views
Trying to block images from google image indexing but not working
I am trying to block images from google image search, but for some reason they are not working. I have a Wordpress self-hosted blog and I have full access to the backend. HTML of mydomain.com looks ...
- 21
8
votes
1
answer
791
views
Are 'robots.txt' rules "starts with" or "contains" rules (clarification on confusing Google documentation)
Searching for robots.txt pattern matching rules (inc. the non official Google rules), I found this Google dev page where it says
the /fish rule matches any path that starts with /fish (...) it doesn'...
- 183
1
vote
1
answer
281
views
Googlebot Spam / DDoS?
I recently saw a significant googlebot crawl on my website. We're talking 10-20 requests per second. It looked like googlebot, and seemed to be legit googlebot IPs, but the requests had query params ...
- 111
0
votes
0
answers
294
views
Google Search Console diplaying my URLs are indexed but it is not showing in search result by parameter site: search [duplicate]
Can anyone provide the solution about this issue:
4
votes
2
answers
3k
views
Why are Bing and SEMRushBot ignoring crawl-delay in my robots.txt file?
We host a large number of sites that have a large number of catalog pages. We would like to slow down several bots because the traffic is quite excessive from those bots. Specifically, we get quite a ...
- 141
1
vote
1
answer
57
views
How to place subdomain in the sitemap
A site made this sitemap for me:
<urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http:/...
- 11
34
votes
3
answers
10k
views
Is there a difference between an empty robots.txt and no robots.txt at all?
On a webserver I now have to admnistrate I noticed that the robots.txt is empty. I wondered if there's a difference between an empty robots.txt and no file at all.
Do crawlers behave differently in ...
- 453