Questions tagged [web-crawlers]

A computer program that accesses web pages for various purposes (to scrape content, to provide search engines with information about your site, etc.)

Filter by
Sorted by
Tagged with
5
votes
11answers
17k views

Is this Google proxy a fake crawler: google-proxy-66-249-81-131.google.com?

Recently I discover that some variants of a google proxy visits my sites. I doubt these are legal Google crawlers because these crawlers are NOT always behind a proxy (like the hostname describes) and ...
0
votes
0answers
11 views

How to separate human from non-human visits in access logs? [closed]

I need to perform usage analysis on a few websites based on no more than standard access logs, e.g. Apache's access_log. Entries have a timestamp, a URL path, a client IP address, a HTTP status, and ...
0
votes
1answer
35 views

Is it possible to tell web crawlers to agree to a site's TOS before indexing?

Is there a way with robots.txt or page headers to inform the managers of web crawlers they must agree to a terms of service prior to indexing any crawled pages?
3
votes
1answer
35 views

Link rel=“nofollow” vs robots.txt disallow for “/search” URLs?

There's a /search path on my web app that I don't want robots to crawl or index. What would be the best approach for this scenario? 1: Block them from accessing by adding a Disallow for /search on ...
0
votes
0answers
8 views

Does bit.ly pass info about referrer? [duplicate]

So that I will be able to see where people that used my bit.ly link come from? I already checked it with Chrome dev tool in Network tab but there is no info.
19
votes
3answers
5k views

Is it possible to slow the Baiduspider crawl frequency?

Much has been made of the Baidu spider crawl frequency. It's true: "Baiduspider crawls like crazy." I've experienced this phenomenon at sites I work with. In at least one instance, I've found that ...
2
votes
1answer
26 views

Will web crawlers be able to crawl nested sidebar menu for mobile web

I have a simple question. One clicking on any of the breadcrumbs, an overlay appears as per design. For example, in the above image clicking on Physics will bring up all subjects to choose from and ...
0
votes
1answer
19 views

Thousands of visits from Huawei ASN, Singapore. Should I worry about them?

I've got a website with information I do not want to be replicated. I'm browsing my visits, and found thousands of visits per day from AS136907, an ASN of Huawei, Singapore. Always with similar IP ...
3
votes
0answers
44 views

Using display: none VS a SPA approach, when it comes to SEO and performance

Say I have a component like, // first approach (SPA) const Component = () => { const [show, toggle] = React.useReducer(show => !show, false) return ( <div> {show && &...
15
votes
4answers
2k views

Prevent malicious bots from posting spam

I remember a site closed due to misuse and I wonder if bots have a part of it. If the bot is POSTing something to my site what are ways I can combat it? I was thinking of setting some cookies and ...
0
votes
0answers
9 views

What are the steps needed to make my website listed on sitelinks? [duplicate]

I am currently developing a website where i wanted my website to list on sitelinks and i know google's algorithm list the sitelinks. but i wanted to know what are the steps i do to make list my ...
10
votes
2answers
2k views

Should I publish an empty ads.txt file for a site that does not run any ads?

I have seen a few 404 errors when ads.txt is requested on some of the sites that I manage. I Guess the answer may be identical to the question about an empty robots.txt file or no file at all (just ...
4
votes
1answer
170 views

How do I “noindex” a text (.txt) file?

I have a directory full of .txt files that are used as part of a larger javascript application. The text files are not inteded to be directly accessed, and are used as includes on separate pages as ...
2
votes
1answer
76 views

Remove large amounts of pages from Bing Index Explorer

I have a website that I took over in my business. We redid the whole website (which was previously a mix of ASP and HTML) and is now a WordPress site running on nginx. I submitted the sitemap to ...
0
votes
0answers
3 views

Can google index site with js? [duplicate]

I want to start optimizing and SEO a website that seems render in client side. in "view page source" I just see links for CSS and Js. none of the html tags are there. Can Google index the ...
1
vote
1answer
49 views

Blocking bad bots based on useragent in apache conf

I am proposing blocking bad bots using the following code in /etc/sites-enabled/site.conf and I have a few questions. RewriteEngine On RewriteCond %{HTTP_USER_AGENT} useragent1|useragent2|useragent3 [...
0
votes
0answers
29 views

Google search says No information is available for my GitHub page

Today I search my GitHub page in Google Chrome and Firefox (Through Google search) and for the first time, it says no information is available (it says 2019- GitHub under that it says no information ...
0
votes
0answers
17 views

Why Google Cache Showing 404 on my React Base Webpages?

My web page is client-side rendered with React. On hitting the cache, it is showing 404 instead of a rendered page. Example Screenshot When I inspect the Page URL on Search Console, Its parsing and ...
8
votes
1answer
988 views

Googlebot encountering JavaScript errors due to rendering outdated HTML files with newer JS files

We recently encountered an issue where we added a JavaScript variable (e.g. var GLOBAL_VAR = true;) to an HTML page (e.g. /Search/Index) and updated the separate JS file (e.g. /Scripts/search/index.js)...
2
votes
5answers
4k views

Google crawl speed — how fast can it go?

I have a huge website with 5 million pages. Currently Google indexes about 10,000 pages per day. This is very slow, I still have lots of pages that I can't get indexed. Does anyone know what is the ...
3
votes
1answer
70 views

List of potential origins of host domains through which google bot crawls?

What is the potential list of origins of the host domains through which google bot crawls? e.g.: Like Google cache uses the following domain to crawl the APIs: http://webcache.googleusercontent.com/ I ...
0
votes
1answer
47 views

Google not indexing product pages of my website [duplicate]

After many months and many changes to sitemaps/pages trying to solve the problem, Google still is not indexing the product pages of this site (link removed). Here's the sitemap i'm using (link removed)...
1
vote
1answer
27 views

Website crawler returns hundreds of 404 errors for pages that do not exist containing duplicate directories

I just completed a website crawl on my site to vet SEO, which should return 119 pages. The results are around 490, with errors being returned showing hundreds of 404 pages with strange URLs. These ...
1
vote
2answers
32 views

How do irrelevant parts of restructured URL affect SEO?

I'm new to SEO and not sure how exactly google crawler works If I use the following format for URL: example.com/category/id/name The content of the page is loaded by only the id part, the other ...
0
votes
1answer
34 views

What is the best practice for old pages that no longer exist on an Angular site?

We built our eCommerce web site using Angular as a single page application. We have implemented a spinner that runs until the page is responding with the expected content. We have some category pages ...
0
votes
0answers
8 views

infinite loader with console error on a page will harm SEO [duplicate]

I have a single page application where some of the category and brand pages are infinite load on the screen (those pages are older now no more available) I want to avoid them. should I use 404 not ...
-1
votes
1answer
13 views

Why is Azure pinging my site?

I have noticed more and more, traffic from IPs assigned to Microsoft Azure/Microsoft Corporation hitting my WordPress site many times a day. It's direct traffic to just the home and privacy policy ...
0
votes
2answers
43 views

spider my website for a line of code

I want to find out which pages on my website (I use blogger) have a specific line of code. For example, I want to know which pages contain the code: "div class='sss'" Does anyone have an ...
1
vote
1answer
50 views

I have 2.1 million pages, how to get crawled by googlebot more frequently? [duplicate]

I have a project which aggregates public information on companies, startups, founders etc. for a total of >2 million tracked entities worldwide. Yes, every single page is high quality, manually ...
1
vote
2answers
42 views

Does crawler index duplicate page which has canonical URL?

Suppose I have two pages with the same content: example.com/a example.com/b so I put a canonical tag <link rel="canonical" href="example.com/a"/> at page example.com/b. I ...
2
votes
1answer
61 views

What is difference between robots.txt, sitemap, robots meta tag, robots header tag?

So I am trying to learn SEO and I am honestly confused and have following 8 questions. Do I tell a bot not to visit a certain link through X-Robots-Tag or through robot meta tag or robots.txt? Is it ...
1
vote
1answer
51 views

Can I access the old version of Google Search Console to see crawling and indexing?

I remember google webmaster console had a crawl and indexing, But in new version, I couldn't find crawling or indexing. Everything is different. Can I access to old style of google console?
2
votes
2answers
156 views

Disallow root but not 4 subdirectories for robots.txt

I have a project and I would like to disallow everything starting with root. From what I understand I think I can do so by doing this Disallow: / Disallow: /* However I would like to allow 4 ...
2
votes
1answer
46 views

Why don't SEO tools such as Screaming frog crawl some of my links?

I have several internal links in my inner pages, for example Interlaken, Switzerland. I used several SEO tools including Screaming Frog to generate sitemap. All SEO tools index links under the "...
4
votes
2answers
3k views

Ampersand (&) in actual URL and sitemap

I have a sitemap (which is submitted to Google for indexing) in which I am adding URLs which have an ampersand (&) in it. Since in sitemap & has to be escaped I replaced & with &amp; ...
1
vote
2answers
106 views

Why are Bing and SEMRushBot ignoring crawl-delay in my robots.txt file?

We host a large number of sites that have a large number of catalog pages. We would like to slow down several bots because the traffic is quite excessive from those bots. Specifically, we get quite a ...
10
votes
2answers
17k views

Only allow Google and Bing bots to crawl a site

I am using following robots.txt file for a site: Target is to allow googlebot and bingbot to access the site except the page /bedven/bedrijf/* and block all other bots from crawling the site. User-...
0
votes
2answers
20 views

What is the time interval that reputable search engines can hit pages of your website during a crawl?

I'm trying to implement some protection against automated attacks on my website (mostly via scripts, probing for SQL injection, etc.) So I'm checking if two consecutive visits from the same IP fall ...
1
vote
1answer
90 views

SEO for blog with async content

I have a blog website and I want to perform Search Engine Optimization on it. My website, which is built in Angular 8, takes part of the URL route and uses that to search for the blog. For example, ...
1
vote
0answers
17 views

Disallowing a handler in robots.txt while adding its dynamic URLs to the XML sitemap

I'm using ASP.NET webforms, and I have a page lets call it Subjects.aspx, I don't want crawlers to crawl that page but I want them to crawl the dynamic URLs that are powered by it. For example /...
2
votes
1answer
41 views

Form-locked PDFs in JSON-LD

I'm setting up downloads functionality with form-locked PDFs - available after the user fills-out a form. I want the PDFs to be indexable by search engines though. Im trying to create a JSON-LD markup,...
2
votes
1answer
47 views

Robots.txt for a multilanguage site where root is redirected

I have a site which offers two languages, English and Spanish. When the user navigates to the home page, let's say www.example.com the page redirects you to either /es if your browser language is ...
0
votes
0answers
54 views

Is it SEO problematic to create footnotes with frontend JavaScript?

I have an RTL (Hebrew) MediaWiki 1.34.0 website in which I am having a serious problem creating footnotes for articles with the core Extension:Cite because its de facto lack of support in Hebrew, as ...
1
vote
1answer
45 views

Google image crawler won't respect my robots.txt entry to not crawl images

I was looking for a way to prevent reverse image searching (namely I didn't want for people who had a copy of one of my images to upload it to google and discover where it originated from). I created ...
1
vote
1answer
116 views

Too many hits from GoogleBot

The last week I have about 500.000 hits/day from googlebot in one of my websites. As a result, the server has serious issues to handle the rest of the requests. What I have done Verified that it is ...
3
votes
1answer
60 views

Can robots.txt redirect to a different domain?

I would like to know if I can put a 301 redirect on robots.txt file and have it hosted in a different location. I'd like to have it on a cloud front layer. example: https://www.mydomain.example/...
1
vote
2answers
32 views

How will Google treat my site if 90% of the URLs redirect and the internal links still point to the old URLs?

Our newly redesigned website version has so many redirects, because there were so many structural errors, SEO issues, and thin content. 90% of the URLs on the Website redirect. Total number of ...
0
votes
2answers
18 views

How to get rid of unnecessary url's from google search console?

66 pages of my website which are all deleted still appears in page report of google search console. My eagerness is knowing all the possibilities. Questions include: 1. does this affect SEO? 2. How ...
0
votes
1answer
18 views

Bing attempting to index URLs that have never existed

I have several sites with dynamic links to products, where the product attributes are encoded into the URL. e.g. https://example.com/products/{"product_code":"Widget","size":"gargantuan"} https://...
1
vote
1answer
21 views

Will Googlbot and other crawlers see content inserted by JavaScript on the window load event?

Many robots, especially Googlebot, can read content written by JavaScript on sites. There are some methods developed especially for site speed. At the beginning of these, do some operations after the ...

1
2 3 4 5
16