Questions tagged [web-crawlers]

A computer program that accesses web pages for various purposes (to scrape content, to provide search engines with information about your site, etc.)

Filter by
Sorted by
Tagged with
1
vote
1answer
38 views

Too many hits from GoogleBot

The last week I have about 500.000 hits/day from googlebot in one of my websites. As a result, the server has serious issues to handle the rest of the requests. What I have done Verified that it is ...
1
vote
1answer
15 views

Remove large amounts of pages from Bing Index Explorer

I have a website that I took over in my business. We redid the whole website (which was previously a mix of ASP and HTML) and is now a WordPress site running on nginx. I submitted the sitemap to ...
2
votes
1answer
42 views

Ahrefs reports “Fetching robots.txt took too long”

My robots: User-agent: * Disallow: /wp-admin Disallow: /sistema Disallow: /site Disallow: /old Sitemap: http://www.example.com.br/page-sitemap.xml In the google test tool: https://www.google.com/...
0
votes
0answers
6 views

301 redirect to robots.txt file to a different domain , may be to a cloud front layer

I would like to know if i can put a 301 redirect on robots.txt file and navigate it to a different location? example site= https://www.example.net/robots.txt redirects to https://differentdomain.net/...
1
vote
1answer
22 views

How will Google treat my site if 90% of the URLs redirect and the internal links still point to the old URLs?

Our newly redesigned website version has so many redirects, because there were so many structural errors, SEO issues, and thin content. 90% of the URLs on the Website redirect. Total number of ...
2
votes
2answers
125 views

What does crawl-delay: 1 mean if it is interpreted as is?

I have a problem of understanding what crawl-delay: 1 means if it isn't ignored and actually being interpreted as is by a given web crawler. I understood from this answer by Stephen Ostermiller that: ...
2
votes
1answer
68 views

New hostile web crawler from China throttling server

My server has been crashes several times the last 24 hours by a new extremely aggressive hostile distributed web crawler. It is hitting over 1000 pages per minute and seems to be crawling all content ...
1
vote
2answers
124 views

crawl-delay: x ------— has someone here managed to understand how the math works there?

From prior research of reading different articles, I understand that a robots.txt crawl delay such as crawl-delay: 1 allows a given search engine to crawl x number of webpages per one second, and than ...
1
vote
0answers
9 views

When setting up a campaign, Facebook is crawling my website using an Iphone user agent

Whenever I'm I'm setting up a campaing on Facebook, I get two visits from Facebook's auto crawlers. The first one I can identify as a robot and serve the SSR (HTML) version to it. The user agent is: ...
1
vote
1answer
24 views

What are the SEO ramifications of duplicate content on the same page?

My question has some similarities to this one. But since I am asking about a very specific type of section on the page I think it warrants some thought as to whether the approach I am looking to take ...
0
votes
0answers
18 views

bing doesn't index my site.why? [duplicate]

i have been indexed my site to bing webmaster ,i submitted urls manually for many times .everything is fine but i don't know why it doesn't appear in search anything form my site. These two picture ...
3
votes
2answers
108 views

Why is my meta description blank in SERPs?

My client's site, which runs on Nuxt and Vue, is not showing meta descriptions on Google SERPs. They used to and as of last week they don't. Any idea why meta description tags would be stripped from ...
1
vote
1answer
15 views

Is it normal that URL's are getting crawled in Screaming frog Still they have a no-follow and no-index tag on them

I'm getting a lot of URL's in screaming frog where the URLs are no-follow no-index but, they are still getting crawled in screaming frog. Is it fine or should I change anything in this?
1
vote
1answer
66 views

How does Google choose “Desktop” or “Smartphone” for the “primary crawler” of a site?

How is the "primary crawler" selected for a site in Google Search Console (coverage segment)? This picture belongs to 2 different websites:
0
votes
0answers
9 views

How can I created tracking redirects that are neither crawled nor indexed?

I have tracking redirects on my site. I use server-side tracking on them. They are used for things like: Advertisements Social media sharing The URLs are of the forms: /ad-click?id=12345 /...
0
votes
0answers
19 views

How do we stop Google Bot from crawling the paginated pages? [duplicate]

I just don't want to index paginated pages in the search result. I see there are already some pages like this one www.example.com/page/18/ being indexed. Maybe I can try setting a canonical Url but ...
1
vote
1answer
23 views

Hide /path from search engines, except for /path/that-i-want-indexed

In our robots.txt we have something like User-agent: * Disallow: /v6.0 Disallow: /v5.8 Disallow: /v5.7 Disallow: /v5.5 Disallow: /v5.1 Disallow: /v5.0 Which properly prevents search engines from ...
0
votes
0answers
7 views

How to change website display layout in search engines? [duplicate]

Pardon me. My question might be wrong. I didn't know what that thing is called and gave a try. Image below shows search result of a news site and you can see the layout. Site name is at top ...
4
votes
1answer
159 views

Does Google index hidden pages based on direct visits on Chrome?

Suppose I have a "hidden" page on a website, but I don't explicitly tell bots to ignore it (no noindex tag, not disallowed in robots.txt, etc). It's not linked to anywhere, doesn't appear on the ...
3
votes
1answer
77 views

Why are bots repeatedly crawling pages of my site that I never created?

In my logs, I noticed that a lot of requests were coming from different bots and web-crawlers, but what confused me was that they were visiting pages such as /fpss/track/73/ that definitely do not ...
3
votes
1answer
304 views

Does the tag “data-n-head” negatively affect seo?

I have a site I am working on which is built in Nuxt. I believe this tag is Nuxt specific in which case I believe crawlers should ignore. Here is more info on it: https://midstride.com/seo-for-nuxt-...
3
votes
2answers
163 views

How to prevent hotlinking for specific domains using .htaccess?

I'm being heavily spammed by image scraper sites. There are more than 150+ domains targeting my images & sending spammed links to my site. I just want to prevent hotlinking from them. I just want ...
1
vote
1answer
53 views

Why would Google Search Console report many times more indexed URLs than a Screaming Frog crawl finds?

I've just crawled my website in Screaming Frog and I have around 6K Internal URLs and 4K external URLs. Then I looked in GSC and saw that Google indexed 50K pages. How is that possible?
3
votes
1answer
27 views

Googlebot-Image incorrectly crossing domain URLs

My website (www.example.com) has all of my images hosted via CDN under a different sub-domain (cdn.example.com). We've noticed an odd behavior recently, that seems as though Googlebot-Image/1.0 is ...
2
votes
1answer
69 views

Allow all bots except Googlebot which should only be able to crawl the home page

I want to allow all bots to crawl and index a site, except Googlebot. I want to allow google to index only the home page (/) URL, but nothing else. I'd prefer to do this in robots.txt. How can I ...
1
vote
2answers
87 views

Does Google index page URLs with languages parameters as a different page?

If I implement multilingual access to the resource through the parameter: 1) https://example.com/en/article-1 2) https://example.com/article-1?l=en So, does Google index both of them as a single ...
2
votes
1answer
32 views

If a web crawler get a 404 error, will it not index that url?

Say I have a page page.example.com/page. In my (python) server, you only get served html if you are logged in, which you can do at page.example.com/login. If you aren't logged in, the server responds ...
1
vote
1answer
97 views

How long it takes for new images to get indexed by Google in general? (now i'm on a Squarespace site if it makes any difference)

i'm wondering, if there is some benchmark out there on this info? I see many sources say it's about 4 days to 4 weeks to get new sites indexed But it's about general web crawling and not ...
0
votes
0answers
14 views

In a sitemap, should I update the lastmod tag of a url based on the text content or html content? [duplicate]

Imagine I have this blogging / ecommerce website with 1000 posts / products. And I've built a sitemap for it, which is dynamically generated. Basically it's a list with a bunch of <url> and <...
1
vote
0answers
30 views

How can I distinguish users from robots to use server side rendering with all bots?

How can I reliably distinguish users from robots? We have an ecommerce shop that goes through some heavy traffic peaks at some times. We would like to move some/all of the load during these peaks to ...
1
vote
0answers
22 views

Can I Prevent Adsense Loading For GoogleBot?

I have been working on lots of performance updates to my site and I have gotten pretty much every page type to load in under 2 seconds. Most single post pages are coming in at around 1.5 seconds.When ...
2
votes
2answers
38 views

Will lengthening our URLs change how Googlebot crawls our site?

Currently we are using URL fetched from main title of article (55 characters). We are planning to increase title of article up to 100 characters which tends to increase length of URL to 100+. Will ...
1
vote
2answers
44 views

Google Search Console reports it is not crawling the hompage but it shows up in a site: search

I recently retouched a corporate site for a client. They also have an ecommerce version of their site at another URL. I've submitted the sitemap and did a manual re-crawl request for the homepage of ...
1
vote
1answer
24 views

How does a live test in Google search console affect ranking if it doesn't pick up all the images and other content?

In google search console I conducted live test URL for my site's home page. However, the screenshot of the live test does not look like my home page. There are not many images and text. My home page ...
3
votes
1answer
56 views

Does Reverse proxy in IIS effects SEO in any way

I'm creating a website using Angular and have enabled server-side rendering which uses a Node JS server. Our site is deployed on an IIS server. To get content from the Node server we are using a ...
1
vote
0answers
11 views

Google Search Console errors (nofollow?)

I have a web site where some internal links take you to another pages of the same site, passing some parameters. Eg.: <a href="/BookMgm/PreparePayment?quantity=1&itemId=123">Book</a> ...
0
votes
0answers
55 views

Using prerender.io for SEO on my AngularJS app - Dynamic and static web pages

I have a website that was built with AngularJS. I want the website to be crawled by search engines (with the dynamic content). I've tried to use Google services to do the crawl (from a sitemap.xml ...
1
vote
1answer
510 views

Google Search Console: URL is not on Google (page with redirect)

I have just created a personal website which I host as github page and recently hooked it up with a custom domain. (In this post I changed the actual URLs to webmasters-example.com and webmasters-...
3
votes
1answer
167 views

How to change Google crawl rate in new Google search console?

Google has now completely removed the old search console (https://searchengineland.com/the-old-google-search-console-is-no-longer-available-321650) where you used to be able to request a custom crawl ...
1
vote
0answers
26 views

Web page crawling by Google for SEO

On Our website we have an angular component which have multiple tabs and loads all the data (of all the tabs) at once. As per our new requirement to increase the 1st load speed, we will be loading ...
2
votes
1answer
60 views

Can I turn off my website at certain times to save money and schedule when Googlebot crawls to maintain SEO?

Is it possible to schedule Google crawlers' crawling at a time when my website it turned on? I host my website with Google's cloud product: Google Compute. To save costs I am thinking of turning ...
2
votes
1answer
44 views

Override robots.txt exclusion to allow Mediapartners-Google for AdSense

I use robots.txt to block all robots.txt compliant bots from a directory like this: User-agent: * Disallow: /sub/* How can I override this so to allow Mediapartners-Google (AdSense) to /sub/?
4
votes
1answer
54 views

Can it be destructive to disallow all MediaWiki:Special pages in robots.txt?

I consider preventing indexation of all of MediaWiki 1.33.0's Special pages; (In Hebrew, "מיוחד" means "special"): Disallow: /מיוחד:* Disallow: /index.php?title=מיוחד:* Disallow: /index.php%3Ftitle%...
3
votes
1answer
64 views

Do search engines crawl JavaScript generated pages?

I'm considering building a new website using react.js and am doing quite a bit of research before embarking into anything. I have found quite a few articles stating that websites that have content ...
2
votes
1answer
47 views

Will this robots.txt file work? [closed]

User-agent: * Allow: /sitemapindex.xml Allow: /post_google_news.xml Allow: /news/ Allow: /category/news/ Disallow: / I want to exclude everything from being indexed by Google except for example.com/...
1
vote
1answer
37 views

SEO for blog with async content

I have a blog website and I want to perform Search Engine Optimization on it. My website, which is built in Angular 8, takes part of the URL route and uses that to search for the blog. For example, ...
1
vote
0answers
29 views

Limit search engines to indexing just one domain when multiple domains share the same web server and document root

I have multiple domains listening on same ports and sharing same document root. This is a limitation of my hosting provider that lets me have multiple domains for my GSuite mail routing without using ...
-1
votes
3answers
577 views

How to remove bulk urls from google index

I have URLs indexed by Google and I want to remove them from indexing. I found that I can use bulk remove option by providing a CSV file including all indexed URLs to Google Search Console. I have ...
3
votes
1answer
81 views

Is it problematic to include both decoded and encoded versions of the same directives in robots.txt?

This question is a follow up to this one: Should one hide RTL encoded URLs in robots.txt or not? Is it problematic to include both decoded and encoded versions of the same directives in robots.txt, ...
2
votes
3answers
73 views

Stop googlebot crawling URL more than once?

I have a site that usually creates a few thousand pages a day, which don't change after they have been created. Recently my dedicated server has crashed due to googlebot crawling the site too often. ...

1
2 3 4 5
16