Questions tagged [web-crawlers]

A computer program that accesses web pages for various purposes (to scrape content, to provide search engines with information about your site, etc.)

Filter by
Sorted by
Tagged with
2
votes
1answer
22 views

Robots.txt for a multilanguage site where root is redirected

I have a site which offers two languages, English and Spanish. When the user navigates to the home page, let's say www.example.com the page redirects you to either /es if your browser language is ...
8
votes
1answer
947 views

Googlebot encountering JavaScript errors due to rendering outdated HTML files with newer JS files

We recently encountered an issue where we added a JavaScript variable (e.g. var GLOBAL_VAR = true;) to an HTML page (e.g. /Search/Index) and updated the separate JS file (e.g. /Scripts/search/index.js)...
0
votes
0answers
41 views

Is it SEO problematic to create footnotes with frontend JavaScript?

I have an RTL (Hebrew) MediaWiki 1.34.0 website in which I am having a serious problem creating footnotes for articles with the core Extension:Cite because its de facto lack of support in Hebrew, as ...
1
vote
1answer
54 views

SEO for blog with async content

I have a blog website and I want to perform Search Engine Optimization on it. My website, which is built in Angular 8, takes part of the URL route and uses that to search for the blog. For example, ...
1
vote
1answer
27 views

Google image crawler won't respect my robots.txt entry to not crawl images

I was looking for a way to prevent reverse image searching (namely I didn't want for people who had a copy of one of my images to upload it to google and discover where it originated from). I created ...
1
vote
1answer
59 views

Too many hits from GoogleBot

The last week I have about 500.000 hits/day from googlebot in one of my websites. As a result, the server has serious issues to handle the rest of the requests. What I have done Verified that it is ...
3
votes
1answer
23 views

Can robots.txt redirect to a different domain?

I would like to know if I can put a 301 redirect on robots.txt file and have it hosted in a different location. I'd like to have it on a cloud front layer. example: https://www.mydomain.example/...
9
votes
2answers
1k views

Should I publish an empty ads.txt file for a site that does not run any ads?

I have seen a few 404 errors when ads.txt is requested on some of the sites that I manage. I Guess the answer may be identical to the question about an empty robots.txt file or no file at all (just ...
1
vote
2answers
27 views

How will Google treat my site if 90% of the URLs redirect and the internal links still point to the old URLs?

Our newly redesigned website version has so many redirects, because there were so many structural errors, SEO issues, and thin content. 90% of the URLs on the Website redirect. Total number of ...
0
votes
2answers
18 views

How to get rid of unnecessary url's from google search console?

66 pages of my website which are all deleted still appears in page report of google search console. My eagerness is knowing all the possibilities. Questions include: 1. does this affect SEO? 2. How ...
0
votes
1answer
14 views

Bing attempting to index URLs that have never existed

I have several sites with dynamic links to products, where the product attributes are encoded into the URL. e.g. https://example.com/products/{"product_code":"Widget","size":"gargantuan"} https://...
2
votes
1answer
19 views

Remove large amounts of pages from Bing Index Explorer

I have a website that I took over in my business. We redid the whole website (which was previously a mix of ASP and HTML) and is now a WordPress site running on nginx. I submitted the sitemap to ...
0
votes
1answer
24 views

How do irrelevant parts of restructured URL affect SEO?

I'm new to SEO and not sure how exactly google crawler works If I use the following format for URL: example.com/category/id/name The content of the page is loaded by only the id part, the other ...
1
vote
1answer
18 views

Will Googlbot and other crawlers see content inserted by JavaScript on the window load event?

Many robots, especially Googlebot, can read content written by JavaScript on sites. There are some methods developed especially for site speed. At the beginning of these, do some operations after the ...
4
votes
2answers
207 views

Does Google index hidden pages based on direct visits on Chrome?

Suppose I have a "hidden" page on a website, but I don't explicitly tell bots to ignore it (no noindex tag, not disallowed in robots.txt, etc). It's not linked to anywhere, doesn't appear on the ...
1
vote
1answer
26 views

A Hash Link should be indexed and crawled

I am building my TOC manually by specifying the anchor and build my bullet list. The links are shown as https://example.com/posturl#test1 Do you think these Hash links should be crawled and indexed ...
18
votes
3answers
10k views

Are web crawlers able to find a secondary robots.txt in a sub-directory?

I have a sub-directory that I would like to hide from web crawlers. One way to do this is to use a robots.txt in the root directory of the server and add a rule to exclude that sub-directory. However,...
1
vote
2answers
124 views

What happens if we forgot to write meta tags?

If we forgot to write meta tags, especially meta title and meta description, how will search engines crawl and index?
2
votes
1answer
127 views

Ahrefs reports “Fetching robots.txt took too long”

My robots: User-agent: * Disallow: /wp-admin Disallow: /sistema Disallow: /site Disallow: /old Sitemap: http://www.example.com.br/page-sitemap.xml In the google test tool: https://www.google.com/...
1
vote
2answers
1k views

Robots.txt isn't preventing my site from being crawled

I'm having problem with robots.txt. I put the robots.txt file in the website main directory (and also in /var/www/html - to make it work on all the server) but robots still keep crawling my websites. ...
10
votes
6answers
15k views

How to properly (dis)allow the archive.org bot? Did things change, if so when?

I have a website that I mostly don't want to be indexed by search engines, but I do want to preserve it for eternity on archive.org. So my robots.txt starts with this: User-agent: * Disallow: / ...
8
votes
5answers
4k views

Deny access to Archive.is

I would like to deny archive.is from having access to my website. (I do not want this website to cache mine without my consent). Do you know if it is possible?
3
votes
2answers
129 views

Why is my meta description blank in SERPs?

My client's site, which runs on Nuxt and Vue, is not showing meta descriptions on Google SERPs. They used to and as of last week they don't. Any idea why meta description tags would be stripped from ...
1
vote
1answer
25 views

What are the SEO ramifications of duplicate content on the same page?

My question has some similarities to this one. But since I am asking about a very specific type of section on the page I think it warrants some thought as to whether the approach I am looking to take ...
2
votes
2answers
160 views

What does crawl-delay: 1 mean if it is interpreted as is?

I have a problem of understanding what crawl-delay: 1 means if it isn't ignored and actually being interpreted as is by a given web crawler. I understood from this answer by Stephen Ostermiller that: ...
1
vote
2answers
124 views

crawl-delay: x ------— has someone here managed to understand how the math works there?

From prior research of reading different articles, I understand that a robots.txt crawl delay such as crawl-delay: 1 allows a given search engine to crawl x number of webpages per one second, and than ...
4
votes
10answers
16k views

Is this Google proxy a fake crawler: google-proxy-66-249-81-131.google.com?

Recently I discover that some variants of a google proxy visits my sites. I doubt these are legal Google crawlers because these crawlers are NOT always behind a proxy (like the hostname describes) and ...
2
votes
1answer
77 views

New hostile web crawler from China throttling server

My server has been crashes several times the last 24 hours by a new extremely aggressive hostile distributed web crawler. It is hitting over 1000 pages per minute and seems to be crawling all content ...
1
vote
1answer
36 views

Form-locked PDFs in JSON-LD

I'm setting up downloads functionality with form-locked PDFs - available after the user fills-out a form. I want the PDFs to be indexable by search engines though. Im trying to create a JSON-LD markup,...
1
vote
0answers
13 views

When setting up a campaign, Facebook is crawling my website using an Iphone user agent

Whenever I'm I'm setting up a campaing on Facebook, I get two visits from Facebook's auto crawlers. The first one I can identify as a robot and serve the SSR (HTML) version to it. The user agent is: ...
3
votes
1answer
28 views

Googlebot-Image incorrectly crossing domain URLs

My website (www.example.com) has all of my images hosted via CDN under a different sub-domain (cdn.example.com). We've noticed an odd behavior recently, that seems as though Googlebot-Image/1.0 is ...
6
votes
5answers
22k views

What does “Disallow: /search” mean in robots.txt?

In my blog's Google Webmaster Tools panel, I found the following code in my robots.txt of blocked URLs section. User-agent: Mediapartners-Google Disallow: /search Allow: / I know that Disallow will ...
0
votes
0answers
18 views

bing doesn't index my site.why? [duplicate]

i have been indexed my site to bing webmaster ,i submitted urls manually for many times .everything is fine but i don't know why it doesn't appear in search anything form my site. These two picture ...
4
votes
3answers
982 views

How can I prevent bad bots from clicking on my advertisers' ads?

I have a website where I create ads for clients. They are not AdWords or similar, they are custom ads I create. All ad links go through a count-clicker before being sent to the advertiser's URL (ex. ...
1
vote
1answer
16 views

Is it normal that URL's are getting crawled in Screaming frog Still they have a no-follow and no-index tag on them

I'm getting a lot of URL's in screaming frog where the URLs are no-follow no-index but, they are still getting crawled in screaming frog. Is it fine or should I change anything in this?
1
vote
1answer
92 views

How does Google choose “Desktop” or “Smartphone” for the “primary crawler” of a site?

How is the "primary crawler" selected for a site in Google Search Console (coverage segment)? This picture belongs to 2 different websites:
0
votes
0answers
9 views

How can I created tracking redirects that are neither crawled nor indexed?

I have tracking redirects on my site. I use server-side tracking on them. They are used for things like: Advertisements Social media sharing The URLs are of the forms: /ad-click?id=12345 /...
0
votes
0answers
19 views

How do we stop Google Bot from crawling the paginated pages? [duplicate]

I just don't want to index paginated pages in the search result. I see there are already some pages like this one www.example.com/page/18/ being indexed. Maybe I can try setting a canonical Url but ...
1
vote
1answer
30 views

Hide /path from search engines, except for /path/that-i-want-indexed

In our robots.txt we have something like User-agent: * Disallow: /v6.0 Disallow: /v5.8 Disallow: /v5.7 Disallow: /v5.5 Disallow: /v5.1 Disallow: /v5.0 Which properly prevents search engines from ...
0
votes
0answers
7 views

How to change website display layout in search engines? [duplicate]

Pardon me. My question might be wrong. I didn't know what that thing is called and gave a try. Image below shows search result of a news site and you can see the layout. Site name is at top ...
3
votes
1answer
2k views

URL will be indexed only if certain conditions are met

I've got a few pages of my site that are coming up in G Webmaster Tools with errors. The pages are fully working, not restricted by noindex or robots.txt file. The pages don'r have a lot of content, ...
3
votes
1answer
82 views

Why are bots repeatedly crawling pages of my site that I never created?

In my logs, I noticed that a lot of requests were coming from different bots and web-crawlers, but what confused me was that they were visiting pages such as /fpss/track/73/ that definitely do not ...
5
votes
2answers
889 views

Delivering a different version of website to Google robots (SSR)

I am working on an enterprise website project as an SEO manager. They were using Angular.js by now but I made them switch to SSR (server-side rendering) since Google robots were not able to render ...
3
votes
2answers
217 views

How to prevent hotlinking for specific domains using .htaccess?

I'm being heavily spammed by image scraper sites. There are more than 150+ domains targeting my images & sending spammed links to my site. I just want to prevent hotlinking from them. I just want ...
3
votes
1answer
510 views

Does the tag “data-n-head” negatively affect seo?

I have a site I am working on which is built in Nuxt. I believe this tag is Nuxt specific in which case I believe crawlers should ignore. Here is more info on it: https://midstride.com/seo-for-nuxt-...
2
votes
1answer
565 views

Two pages have same title, same description, different content. Are they duplicates?

We have release our website from our mobile app in a very short amount of time. The website is quite good, but not at all optimized for SEO. For example, there is absolutely no care for meta tags. All ...
1
vote
1answer
54 views

Why would Google Search Console report many times more indexed URLs than a Screaming Frog crawl finds?

I've just crawled my website in Screaming Frog and I have around 6K Internal URLs and 4K external URLs. Then I looked in GSC and saw that Google indexed 50K pages. How is that possible?
1
vote
1answer
168 views

How long it takes for new images to get indexed by Google in general? (now i'm on a Squarespace site if it makes any difference)

i'm wondering, if there is some benchmark out there on this info? I see many sources say it's about 4 days to 4 weeks to get new sites indexed But it's about general web crawling and not ...
2
votes
1answer
80 views

Allow all bots except Googlebot which should only be able to crawl the home page

I want to allow all bots to crawl and index a site, except Googlebot. I want to allow google to index only the home page (/) URL, but nothing else. I'd prefer to do this in robots.txt. How can I ...
1
vote
2answers
112 views

Does Google index page URLs with languages parameters as a different page?

If I implement multilingual access to the resource through the parameter: 1) https://example.com/en/article-1 2) https://example.com/article-1?l=en So, does Google index both of them as a single ...

1
2 3 4 5
16