Questions tagged [web-crawlers]

A computer program that accesses web pages for various purposes (to scrape content, to provide search engines with information about your site, etc.)

Filter by
Sorted by
Tagged with
2
votes
1answer
19 views

SEO | Cloaking | Is it safe to remove Third-party / Analytical scripts for crawlers/bots? Is this considered as cloaking?

Is it considered as cloaking if I remove third party scripts such as google tag manager, other analytics based scripts tags for crawlers/bots? By removing these external javascripts I will surely have ...
0
votes
0answers
16 views

Crawl and Indexing [closed]

I'm relatively new to blogging, had set up my site, written contents. However, I'm sort of having slight challenges. Google and Bing aren't indexing my site categories Made few adjustments, but ...
1
vote
1answer
77 views

Do I need to allow bots on /* if I disallow /specific-directory?

If I want all bots to not crawl /specific-directory, but I want them to operate "as usual" (thus: crawl) all other directories of my site, is this robots.txt entry adequate? # All Bots User-...
1
vote
0answers
22 views

Does Googlebot respect the cache-control header?

Does Googlebot respect the cache-control header? By setting an appropriate cache-control header, can one improve the crawl efficiency?
0
votes
1answer
47 views

SEO Impact of Vary: User-Agent header

Vary: User-Agent header is used to tell browsers, CDNs, etc., (for caching purposes) that the content differs by user agents (especially for mobile and desktop versions). But do search engine robots ...
0
votes
0answers
24 views

Quickest way to remove many dynamic pages with duplicate content due to a spider trap

I have an angular JS site that is reliant on google rendering the pages. I created a spider trap and have about 9000 pages in the index which most are duplicate content. Each page that is currently ...
0
votes
0answers
17 views

Does Googlebot crawl JS or JSON inside an HTML page?

My Google Search tool is complaining about certain URLs it found on my site. These are intentionally unusable URLs; I'm not using them in any links. Think of them as unused meta-data. A) In one ...
0
votes
0answers
7 views

Block some subdomains using robots.txt -Angular website with IIS [duplicate]

We have a testing environment with the surprising sub-domain name tests and I would like to block it to search engines. How can I let Google scan my site https://example.com and block https://tests....
1
vote
1answer
34 views

How does authentication-required content get indexed/crawled?

Assume we have a VOD platform to watch online videos and it requires authentication to show that video or any kind of content. User has to be signed up and after that, they can access the videos etc. ...
1
vote
1answer
41 views

Why Googlebot is crawling only one page?

I am trying to understand why Google bot is crawling only one page only on my website. At the Google console I registered a small XML sitemap with 9 URLs. Only one URL I marked as a high priority (0.8)...
2
votes
2answers
51 views

Can I indicate (via robots.txt, meta robots or another approach) that one or more queryString parameters should be ignored by crawlers?

I've written my own SiteSearch Script in PHP. The SiteSearch parses two GET parameters from the queryString: search // the search-phrase filters (optional) // indicating which parts of the site to ...
0
votes
1answer
25 views

Google's priority on “My Business” vs Local Business Structured Data

There are at least two ways for providing rich data about business to Google: Using Google My Business: comes with a convenient web UI and is officially run by Google Using Google Structured Data: ...
0
votes
0answers
25 views

In my Crawl Stats, 23% of requests are “Unauthorized” - is that bad?

Am I correct in thinking that having 23% of all crawl requests being blocked by "Unauthorized" 401/407 is NOT good? 68% of my crawl requests return 200 (OK) which is good of course. Is this ...
1
vote
1answer
23 views

Can search engine fetch/index website that using ipv6 only?

I see some host providers offer cheap server but only support ipv6. I want to rent it, but I am still considering the SEO impact. Can google and other search engine crawl and index it? If yes, will it ...
0
votes
0answers
37 views

How to block google from indexing mail record. duplication issue mail.example.com which is duplication of example.com

google is causing duplication by indexing mail.example.com which is identical to home page of www.example.com/ I added property in search console and removed this subdomain but keeps on indexing ...
2
votes
1answer
49 views

Do I need to submit sitemap.xml to google if I already have it in robots.txt and hosted on S3?

I have a static site hosted on S3 bucket which will have a robots.txt at root directory level. This robots.txt file will have reference to my sitemap.xml file which is hosted on another S3 bucket. It ...
2
votes
1answer
40 views

robots meta tag for specific web crawler

Web crawler list What I want noimageindex (Indexation-controlling parameters) will only work for googlebot (2 : two meta tag - code below) Others all crawlers (example as a YandexBot, Baiduspider) ...
2
votes
1answer
36 views

What is a difference in the order of “follow” and “index” in a robots meta tag?

What is the difference between <meta name="robots" content="follow, index"> and <meta name="robots" content="index, follow"> Would either create ...
2
votes
0answers
42 views

Prevent search engines from crawling and indexing all domains and subdomains set up under an Apache server

My company is renting a CentOS server, with one main domain and ~30 subdomains (the number varies, but it's usually around 30) hosted on it. The subdomains themselves are used for various purposes (...
4
votes
0answers
28 views

Consequences of blocking Semrush and other bots?

For typical websites, are there any disadvantages to my clients to blocking spiders like Semrush, Maxmind and the plethora of other "non search-engine" bots? Wouldn't blocking these ...
0
votes
0answers
29 views

What is most likely causing this index:1 404 error in Wordpress?

I'm running the latest version of WordPress on a VPS, and I recently moved my WordPress directory to a new location and reconfigured my site's permalink structure. So when you go to HTTP://www....
0
votes
0answers
13 views

Remove Google suggestions and links from search results [duplicate]

How to remove / delete google suggestions and related keywords links. As some results showing my personal data into some sites , need to remove those pages or links from search results showed by ...
1
vote
2answers
157 views

How to make a bot in Windows that does automated search of domain names to see if they are available? [closed]

Instead of manually typing in domain names in a domain search registry to see if they are available, How to program a bot to run through a pre-specified list of .com domain names by searching each one ...
2
votes
1answer
61 views

Is it legal to use the APIs of Meetup.com, Eventbrite etc, to aggregate their event data into one place, similar to Google?

I have a predicament. I'm trying to build a platform that will basically be a place where all events, online/offline, are listed. Initially - my platform will obviously have zero events, because ...
2
votes
0answers
28 views

Could allowing our images to be indexed hurt our web search rankings?

We are trying to solve some issues related to our SEO. When we compare ourselves to our competitors who are higher that us in SERPS, other than more backlinks (not much more), we don't see any ...
2
votes
0answers
62 views

Restrict Google from indexing subdomain from Azure CDN services

How can I stop Google from indexing example.azureedge.net content? We have the website from last two years but just recently we found out that Google has started to index all the URLs on the azure CDN ...
4
votes
2answers
67 views

How to keep crawler robots from indexing my page when there is a query string present in the URL?

I have a /products page. In that page, users may use filters to select the products they want to see. Those filters will work using query strings like: /products?brands=X,Y&size=1,2 and so on. I ...
0
votes
1answer
31 views

HTTP “Vary: User-Agent” header for a Single Page React App that does SSR for robots?

I have React web app. It's a single page app, so there's basically an "empty" HTML file with a JS bundle that does it all. But to improve SEO capabilities, I'm doing SSR for robot crawlers ...
3
votes
1answer
32 views

Wordpress Websites Getting High Traffic for the non-Existent Pages like a DDOS Attack

I'm facing a weird situation where my 10 WordPress websites are getting hits for non-existent pages. Mostly those pages which we never created or had in our websites. I searched about it and found Non-...
4
votes
2answers
87 views

Bot calling my php script too fast

I have a hobby website, where a php script presents a html page displaying information for a particular surname. I am capable of displaying information for over 100,000 surnames. So the script is ...
1
vote
1answer
31 views

What are Windows NT 10.3 and 10.5?

In my Apache logs, a few malicious scans were done by machines that identify their OS in their user agents as Windows NT 10.3 Windows NT 10.5 Are these fictitious or were there at some point real ...
3
votes
0answers
44 views

Do HTML elements with the boolean attribute hidden get picked up by search engines?

I recently learned of the existence of the hidden attribute. It basically hides elements that are not relevant in the current state of the webpage (e.g., a user that is not logged in): <p hidden>...
0
votes
0answers
13 views

How can I hide my site information in Alexa? Will it be done by blocking the robot? [duplicate]

I try to hide my site information from Alexa or not show some country-specific information. How can I do this? Will this be done by blocking the Alexa robot or do I have to do something else?
6
votes
4answers
3k views

How I can ensure that a link sent via email is opened only via user clicks from a mail client and not by bots?

In my web application I generate links in the following format: https://example.com/^token^ Each link according to my specs is sent via email and should not allow robots to scrape it. In order to ...
1
vote
0answers
21 views

Can you specify the max-snippet length in a X-robots-tag header?

The specs by Google does not address this issue. So, I would like to know what is the x-robots-tag equivalent for: <meta name="googlebot" content="max-snippet:20"/> Is it ...
0
votes
1answer
40 views

Is it possible to tell web crawlers to agree to a site's TOS before indexing?

Is there a way with robots.txt or page headers to inform the managers of web crawlers they must agree to a terms of service prior to indexing any crawled pages?
0
votes
0answers
8 views

Does bit.ly pass info about referrer? [duplicate]

So that I will be able to see where people that used my bit.ly link come from? I already checked it with Chrome dev tool in Network tab but there is no info.
3
votes
1answer
47 views

Link rel=“nofollow” vs robots.txt disallow for “/search” URLs?

There's a /search path on my web app that I don't want robots to crawl or index. What would be the best approach for this scenario? 1: Block them from accessing by adding a Disallow for /search on ...
1
vote
1answer
48 views

Thousands of visits from Huawei ASN, Singapore. Should I worry about them?

I've got a website with information I do not want to be replicated. I'm browsing my visits, and found thousands of visits per day from AS136907, an ASN of Huawei, Singapore. Always with similar IP ...
3
votes
0answers
57 views

Using display: none VS a SPA approach, when it comes to SEO and performance

Say I have a component like, // first approach (SPA) const Component = () => { const [show, toggle] = React.useReducer(show => !show, false) return ( <div> {show && &...
0
votes
0answers
9 views

What are the steps needed to make my website listed on sitelinks? [duplicate]

I am currently developing a website where i wanted my website to list on sitelinks and i know google's algorithm list the sitelinks. but i wanted to know what are the steps i do to make list my ...
4
votes
1answer
232 views

How do I “noindex” a text (.txt) file?

I have a directory full of .txt files that are used as part of a larger javascript application. The text files are not inteded to be directly accessed, and are used as includes on separate pages as ...
2
votes
1answer
32 views

Will web crawlers be able to crawl nested sidebar menu for mobile web

I have a simple question. One clicking on any of the breadcrumbs, an overlay appears as per design. For example, in the above image clicking on Physics will bring up all subjects to choose from and ...
0
votes
0answers
4 views

Can google index site with js? [duplicate]

I want to start optimizing and SEO a website that seems render in client side. in "view page source" I just see links for CSS and Js. none of the html tags are there. Can Google index the ...
0
votes
0answers
44 views

Google search says No information is available for my GitHub page

Today I search my GitHub page in Google Chrome and Firefox (Through Google search) and for the first time, it says no information is available (it says 2019- GitHub under that it says no information ...
1
vote
1answer
71 views

Blocking bad bots based on useragent in apache conf

I am proposing blocking bad bots using the following code in /etc/sites-enabled/site.conf and I have a few questions. RewriteEngine On RewriteCond %{HTTP_USER_AGENT} useragent1|useragent2|useragent3 [...
0
votes
0answers
58 views

Why Google Cache Showing 404 on my React Base Webpages?

My web page is client-side rendered with React. On hitting the cache, it is showing 404 instead of a rendered page. Example Screenshot When I inspect the Page URL on Search Console, Its parsing and ...
3
votes
1answer
71 views

List of potential origins of host domains through which google bot crawls?

What is the potential list of origins of the host domains through which google bot crawls? e.g.: Like Google cache uses the following domain to crawl the APIs: http://webcache.googleusercontent.com/ I ...
1
vote
1answer
48 views

Website crawler returns hundreds of 404 errors for pages that do not exist containing duplicate directories

I just completed a website crawl on my site to vet SEO, which should return 119 pages. The results are around 490, with errors being returned showing hundreds of 404 pages with strange URLs. These ...
0
votes
1answer
145 views

Google not indexing product pages of my website [duplicate]

After many months and many changes to sitemaps/pages trying to solve the problem, Google still is not indexing the product pages of this site (link removed). Here's the sitemap i'm using (link removed)...

1
2 3 4 5
17