Questions tagged [web-crawlers]
A computer program that accesses web pages for various purposes (to scrape content, to provide search engines with information about your site, etc.)
880
questions
1
vote
1
answer
70
views
How did Google crawler view my password-protected pages?
I have a site that is login-blocked/password-protected where Googlebot crawling has worked for 1+years and suddenly stopped.
I am trying to figure out how this could have happened. It is a large site ...
- 171
2
votes
1
answer
36
views
Can I set Cawl-delay in robots.txt to less than one second (like 0.1)?
I would like to start using the Crawl-delay on the robots.txt and saw that the value is in seconds. Is there a way to set it up below 1 second. Maybe 0.1?
My goal is to allow crawlers not to run over ...
- 121
0
votes
0
answers
13
views
Adding Custom robots.txt in blogger
I was looking at examples to add the file to my blogger website and came across a code:
User-agent: Mediapartners-Google
Disallow:
User-agent: *
Disallow: /search
Allow: /
Sitemap:https://example.com/...
2
votes
1
answer
94
views
What does User-agent: Mediapartners-Google mean?
I was going through the google support website and there they said:
"To update your robots.txt file to grant our crawler access to your pages, remove the following two lines of text from your ...
0
votes
1
answer
49
views
Why website copy tools like Cyotek WebCopy and HTTrack cannot find files that search engines like Google can? [closed]
I would like to keep the target website private, but here are some details:
It's a personal (as in single-author) public documentation / portfolio / blog sort of website
It seems to be hosted using ...
- 141
0
votes
0
answers
24
views
Alternate Canonical Tags error on Google Search Console
I recently started my blogging site and after a few days on Google Search Console I got an error saying alternate page with proper canonical tag. Now I searched the internet and found the fix. A line ...
2
votes
1
answer
41
views
How to make a text website hostile only to machine-learning models while maintaining smooth experience for humans?
So, we're building a website that showcases some niche information and tries to present it in a way to build up a knowledge base. We want to maximize accessibility to people and spider bots, while ...
- 23
2
votes
1
answer
61
views
Do article pages need to have an RSS Feed page?
I am working on a blog software. This blog has a home page, category pages and articles.
I will add an RSS feed to this blog. I added the RSS feed to the home page by adding /feed to the end of the ...
- 789
3
votes
1
answer
82
views
What is the actual HTML rendered to bots?
We have a product page in which 4 item carousels are present, showing mostly related products. All these 4 carousels are client side rendered.
1. Scenario1: Google rich results
Now, we have observed ...
- 141
0
votes
1
answer
27
views
200 website code by Console [closed]
Can someone help me and let me know why google is not ranking quite some few of my sites from mine latest SEO project:
https://kroatien-ferienwohnungen.at/
I see that plenty of pages are crawled but ...
- 3
1
vote
1
answer
35
views
Is blocking indexing and crawling at the same time bad?
have a lot of pages that I want to disallow crawling and indexing on, because of crawl and index budgets. I can't just disable crawling because they are linked together with internal links and I can't ...
- 65
3
votes
2
answers
32
views
Prevent search engines indexing admin and other pages
My website has an Admin area which requires a login. There are also other pages that should not be indexed. I know that Robots.Txt entries or meta name="robots" content="noindex" ...
0
votes
1
answer
18
views
Allow and dissallow directory with # in robots.txt [duplicate]
I'd like to block and unblock following URL formats:
Allow:https://example.com/buyer/#/search/catalog/Lubricating-Oils/Automotive-Lubricants
Dissallow:https://example.com/buyer/#/search/product-...
5
votes
2
answers
1k
views
How are 'scraped content' websites like diningandcooking.com able to rank so well despite having no original content?
I've stumbled upon a website called https://diningandcooking.com after finding out it was linking to one of my websites. When i went to the page that had the link on it i found out that its content ...
- 381
2
votes
1
answer
40
views
Is disallowing all useragents sufficient to keep a site from being crawled by search engines?
We have a non-production version of our site that we want to prevent from being crawled. At some point in the past a non-production site was crawled and we still have search results pointing to it. I ...
- 123
2
votes
0
answers
41
views
One day after putting my site online and registering it with Google I got spam mail. How?
I put my website online yesterday and registered it with Google at the same time. Before that, I didn't even put a single link to the site anywhere online. And I don't have any email address on my ...
0
votes
0
answers
66
views
Google Search Console diplaying my URLs are indexed but it is not showing in search result by parameter site: search [duplicate]
Can anyone provide the solution about this issue:
1
vote
1
answer
67
views
Can I Improve my website's Google rankings with a bot that searches for my site and clicks on it?
I have started a new website that I want like to obtain worldwide traffic for. And ofcourse, a key element is how to improve my Google search engine rank.
I am considering to develop a bot that would ...
- 143
0
votes
0
answers
12
views
Why does Google offer an index page in a language other than the main language?
Our website has several language mutations, the basic language is Czech at https://www.example.com/. Other languages are at the addresses English - https://www.example.com/en/, Polish -https://www....
- 101
2
votes
0
answers
27
views
Googlebot increased the number of requests for a certain URL even though it has no links
As part of regular Log File Analysis, I noticed in a URL that Google_bot was crawling like crazy. This URL is an Iframe that embed in a page. My thought was maybe this page gets a new link from ...
- 51
3
votes
2
answers
175
views
How to configure robots.txt to disallow any URL/file/directory with a certain word in it?
I've seen basic articles online about allowing/disallowing certain subdirectories in robots.txt, but I have a more detailed need than that.
Google search console keeps trying to discover/crawl certain ...
- 45
2
votes
0
answers
95
views
What is correct way to stop query indexing using Robots.txt?
How to stop indexing search result using Robots.txt?
hello.
Recently, I'm having an issue with indexed pages in Google Search Console growing 10x.
These pages are mostly search query (/?s=) and ...
- 21
2
votes
2
answers
194
views
Will Googlebot click on tab navigation / buttons and index links?
I have the following UI:
There are tabs which are <button>'s and when you select one there will be <a>-Links to different pages. It seems like Google will only index the Links on the ...
- 143
4
votes
3
answers
516
views
Bing refuses to index my site at all
I have been trying for months now to get Bing to index my personal website, to no avail. I'm not talking about ranking, as I expect it to rank quite low. It is invisible. I don't care about Bing in ...
- 175
1
vote
1
answer
53
views
Will Google crawl links with other nested HTML elements?
I have a link like this
<a href="/some/relative/link">
<div class="container">
Some Content
</div>
</a>
On the Google Search Console, it says that ...
- 113
-1
votes
1
answer
537
views
Does googlebot crawl the variable name of a php [closed]
Does Googlebot crawl the dynamic variable name aside from its value? For example in my PHP I use the variable _company_name. In the front-end is is _company_name saved in bot crawl also?
- 27
0
votes
0
answers
66
views
Bingbot issues and crawling my website root domain [duplicate]
I seem to have this issue with a few different website's but bing is unable to crawl (or fetch the pages).
I have done a cURL via windows and it connected properly however every URL live inspection ...
2
votes
1
answer
29
views
How to modify google selected canonical URL when the URL is not valid anymore
I am the owner of two domains, domain1.com and domain2.com,
A canonical URL : domain2.com is selected by google for domain1.com while the url does not exist anymore, how to make Google re-select ...
- 21
2
votes
1
answer
42
views
Can we request google to crawl our website every single day to update its indexes and tags on our domain?
I m thinking of making a blog website and noticed that google search results appear as per Google's indexing upon the crawled website. And this web crawling by google happens like once in 2 weeks to a ...
- 21
2
votes
1
answer
292
views
Will redirecting to a mailto URL prevent spam bots from harvesting the email address?
On https://www.heartinternet.uk/blog/15-ways-to-hide-your-email-address/, under the sub heading Replacing with a PHP script, a method to hide email addresses is presented that I like very much. On the ...
0
votes
1
answer
43
views
Do Webcrawlers go to multiple subpages with no internal links?
I run a small business, for discussion let's say I have 4 pages:
Page A (Home)
Page B
Page C
Page D.
Important: NONE of these Pages have internal links to each other. Imagine just 4 tabs at the top ...
1
vote
0
answers
30
views
Exposing content from client-side headless CMS API to web crawlers
This is something I've been struggling to wrap my head around for the past few days. Tried to keep the rambling to a minimum but concise questions are at the bottom.
As I understand it, utilizing JS ...
1
vote
1
answer
124
views
Only allow crawling of pages in sitemap
My idea is, to use robots.txt to disallow everyting and put all allowed pages into the sitemaps.
User-agent: *
Disallow: /
Sitemap: https://www.example.com/sitemap.xml
Is this working as expected?
- 11
1
vote
0
answers
104
views
Do Baidu and/or Yandex publish the current IP ranges used by their site indexing bots?
I need to identify bots operated by search engine operators to index sites.
I've looked at an existing question on this matter and the answers given there are too simplistic.
The User-Agent header is ...
- 125
1
vote
1
answer
253
views
Google Indexing Tons of Non-Existent URLs
A client that I am working for has come to me with a problem. Their Google Adwords account has been disabled because there is a bunch of traffic to crazy non-existent pages on their website. Here are ...
- 13
0
votes
1
answer
42
views
Ratio of real visitors to search engine views
I know it is highly website-dependent, but is there any statistics/study/report on the ratio of real visitors websites get and the visits made by the search engine crawler?
Services like Alexa reports ...
- 452
8
votes
1
answer
708
views
Are 'robots.txt' rules "starts with" or "contains" rules (clarification on confusing Google documentation)
Searching for robots.txt pattern matching rules (inc. the non official Google rules), I found this Google dev page where it says
the /fish rule matches any path that starts with /fish (...) it doesn'...
- 183
1
vote
2
answers
427
views
Can a Web Crawler follow links from <Link> tag in React?
I am building a frontend website and I am trying to make sure that the website can be crawled by search engines so that it can appear in search results.
Currently, most of my links look like this in ...
mastercooler6
2
votes
1
answer
52
views
How to use Subdomain for a custom CMS of the main domain website?
Should I use a subdomain for a custom CMS of the main domain website? Like cms.domain.com. As it will be a CMS, I will disallow a search crawler for indexing. Will this hurt SEO for the Main Domain?
4
votes
1
answer
188
views
Is there a way to allow ad bots to crawl a website that has "noindex,nofollow" on it?
Due to duplicated content, there is a "noindex,nofollow" on the website, but in order to run ppc ads, there needs to be a way for the ads bots to crawl the website
Can this be done without ...
- 41
2
votes
2
answers
294
views
Should I exclude the assets folder from search engine crawlers in angular
I'm working with angular and had added the assets folder as element to exclude from the search engine crawlers, into my robots.txt file.
But I now get the following error while testing the site health ...
- 163
2
votes
0
answers
37
views
Disallow crawling of all links to a domain, except only specific URLs with parameters
I have a domain like example.com under which everything is blocked and links to an SPA.
Like example.com, and we've disabled everything like so:
User-agent: *
Disallow: /
Now this SPA uses parameters ...
2
votes
0
answers
145
views
How can I let Google crawl my site for Programmable Search Engine (custom site search), but not have it show in public search results?
The answer for my question might be inherent in the question itself- that if I want Google to crawl my site for a custom site search, such as through Programmable Search Engine, that it is those same ...
- 121
3
votes
2
answers
49
views
Should we skip extra scripts (trackers, CRM, ..) when page is crawled by a bot?
We can identify bots with their User-Agent (1, 2), but that's not so straightforward. The benefit is to skip large and probably useless resources for bots, and just serve core content
What's the best ...
- 85
14
votes
8
answers
5k
views
Is it possible to keep the existence of a domain secret?
If I bought a domain like a9j47fn83jd8j49.tld, and only a friend and I ever visit it, who would know about it? It sounds like for most TLDs, there is some reporting that has to happen after you buy a ...
- 243
3
votes
2
answers
627
views
Does submitting sitemap.xml once change the way the site's new pages will be crawled/discovered in the future?
My static site doesn't generate sitemap.xml automatically.
Hundreds of pages have been modified recently (meta description, etc.). I would like Google to recrawl the whole website. Submitting the URLs ...
- 552
3
votes
1
answer
72
views
How do Google and Yahoo fetch content for pages behind a paywall or registration wall?
How do Google and Yahoo fetch content for pages that are behind a paywall or registration wall? I noticed that search results can show content that is not visible publicly without a paid or registered ...
- 33
3
votes
1
answer
379
views
Is it okay to have hyphens in brand/company name?
My domain name is without hyphens, but I promote/market my brand/company name with hyphens in between (example: 'company-name'). Is it okay to have such difference? Or will it affect in any way to SEO ...
- 131
2
votes
2
answers
110
views
Will bots crawl my page generated by javascript
I use simple javascript to generate pages based on thousands of products stored in a JSON object.
The page URLs will be like /productList?keyword=XXX&page=x. The keyowrd is what uses input through ...
- 495
11
votes
4
answers
6k
views
Prevent Google from thinking my own bot is faking traffic on my website
I have a large website which I want to be visited by my own bot early on the morning, every day (for caching purpose). I want my bot visits 10,000 pages per day.
My natural traffic is about 5000 pages ...
- 111