Questions tagged [web-crawlers]

A computer program that accesses web pages for various purposes (to scrape content, to provide search engines with information about your site, etc.)

Filter by
Sorted by
Tagged with
0
votes
0answers
10 views

MY wordpress website cannot be indexed by google [duplicate]

I have been trying to index my newly created website on wordpress for some time. It was released in early December and only two pages have been indexed. Overall, the site has 9 pages and only two have ...
1
vote
0answers
19 views

How can I let Google crawl my site for Programmable Search Engine (custom site search), but not have it show in public search results?

The answer for my question might be inherent in the question itself- that if I want Google to crawl my site for a custom site search, such as through Programmable Search Engine, that it is those same ...
3
votes
2answers
46 views

Should we skip extra scripts (trackers, CRM, ..) when page is crawled by a bot?

We can identify bots with their User-Agent (1, 2), but that's not so straightforward. The benefit is to skip large and probably useless resources for bots, and just serve core content What's the best ...
14
votes
8answers
5k views

Is it possible to keep the existence of a domain secret?

If I bought a domain like a9j47fn83jd8j49.tld, and only a friend and I ever visit it, who would know about it? It sounds like for most TLDs, there is some reporting that has to happen after you buy a ...
2
votes
2answers
237 views

Does submitting sitemap.xml once change the way the site's new pages will be crawled/discovered in the future?

My static site doesn't generate sitemap.xml automatically. Hundreds of pages have been modified recently (meta description, etc.). I would like Google to recrawl the whole website. Submitting the URLs ...
3
votes
1answer
35 views

How do Google and Yahoo fetch content for pages behind a paywall or registration wall?

How do Google and Yahoo fetch content for pages that are behind a paywall or registration wall? I noticed that search results can show content that is not visible publicly without a paid or registered ...
3
votes
1answer
40 views

Is it okay to have hyphens in brand/company name?

My domain name is without hyphens, but I promote/market my brand/company name with hyphens in between (example: 'company-name'). Is it okay to have such difference? Or will it affect in any way to SEO ...
2
votes
2answers
86 views

Will bots crawl my page generated by javascript

I use simple javascript to generate pages based on thousands of products stored in a JSON object. The page URLs will be like /productList?keyword=XXX&page=x. The keyowrd is what uses input through ...
11
votes
4answers
6k views

Prevent Google from thinking my own bot is faking traffic on my website

I have a large website which I want to be visited by my own bot early on the morning, every day (for caching purpose). I want my bot visits 10,000 pages per day. My natural traffic is about 5000 pages ...
5
votes
1answer
586 views

Why does it cause issues to use robots.txt and a meta robots tag at the same time?

It seems common wisdom that using X-Robots-Tag/meta robots with robots.txt to block a URL from being indexed "can cause issues", e.g.: Using both X-Robots-Tag and meta robots on a URL is ...
4
votes
1answer
163 views

Does google follow links discovered in JSON-LD?

Lets say I have a site with a page of 20 products which are fed through an API. The products are loaded using infinity scroll so the initial page render has 5 products tops. Upon scrolling it adds 5 ...
4
votes
2answers
85 views

Is there any way I can defend against automated file download bots?

My company offers a very specifically tailored Android application to supplement our other software. Since users must have an account in our other software in order to use the Android application, is ...
4
votes
1answer
64 views

Weird bot requests in logs

I restored a website on a new server after it was down (mistakenly deleted) for a while, and it immediately got bombarded with requests to non-existing pages like /tissue-salts-for-weight-loss.html ...
1
vote
1answer
19 views

Robots.txt not blocking yet GSC URL Inspection Tool says it is

Recently I realized Disallow /was set on my staging sites. I didn't realize until I got a warning in Search Console... Mistakes happen I've removed the directive and cleared all caches at least 10 ...
4
votes
1answer
139 views

Do SEO analysis bots have any advantage in terms of website traffic?

I have loads of traffic from SEO analysis bots (which are crawling for link analysis purpose only, rather than indexing for search engines) such as AhrefsBot/7.0; +http://ahrefs.com/robot/ SemrushBot/...
2
votes
1answer
29 views

Are URLs with fragments or paths better for SEO?

Suppose I have a food blog (say www.myfoodblog.example) with several recipes (mango-icecream, banana-icecream, and berry-icecream). So what I understood for SEO is to have these keywords of recipes in ...
0
votes
1answer
30 views

Can I tell who is accessing archive.org archived documents from my site from my server logs?

I currently host a website with a collection of some of my projects, where a few of them are currently undergoing a patent review process. I don't want these documents to be public (though they were ...
1
vote
1answer
28 views

Should I add links to the Sitemap that are disallowed in robots.txt?

My robots.txt contains the line Disallow: /search.php. Shall I nevertheless include http://example.com/search.php into my sitemap?
2
votes
1answer
76 views

What is a robots.txt file and how can I use it

I want to use a robots.txt file for my website, which is currently displaying a 404 error when requesting it (e.g., https://www.example.com/robots.txt). How do I make one? There's no option to create ...
1
vote
2answers
64 views

How to include URLs generated from the WordPress API in my XML sitemap?

I have a website build in Angular 8, which among other things has a blog section. I have a sitemap.xml file that contains all the pages except the blog. The blog is created on WordPress so in order to ...
2
votes
2answers
96 views

What impact do text-based accordion elements have on SEO?

I'm looking for some consensus on what the current school of though is about accordion elements. Here is a page that uses them as an example: https://www.goaskeve.com/venues/goldsmiths-centre/ Will ...
0
votes
1answer
115 views

HTML sitemap with noindex, follow: Matt Cutts vs John Mueller

Ref: Do noindex,follow pages pass pagerank / link juice? I have a requirement wherein I want to create an HTML sitemap to help users navigate to deep pages. This sitemap page will be linked from ...
2
votes
1answer
25 views

How do rule length and use of wildcards determine whether allow or disallow takes precedence in Robots.txt?

I refer to this webpage: https://developers.google.com/search/docs/advanced/robots/robots_txt Could someone elaborate on the difference between these two: http://example.com/page.htm allow: /page ...
2
votes
1answer
41 views

SEO | Cloaking | Is it safe to remove Third-party / Analytical scripts for crawlers/bots? Is this considered as cloaking?

Is it considered as cloaking if I remove third party scripts such as google tag manager, other analytics based scripts tags for crawlers/bots? By removing these external javascripts I will surely have ...
0
votes
2answers
30 views

How to know when a new website comes into existence?

Is there a way to know when a new website comes into existence. What I mean is how crawlers such as search engines know if something exists in first place. So let's say a new website www.example.com ...
1
vote
1answer
86 views

Do I need to allow bots on /* if I disallow /specific-directory?

If I want all bots to not crawl /specific-directory, but I want them to operate "as usual" (thus: crawl) all other directories of my site, is this robots.txt entry adequate? # All Bots User-...
1
vote
0answers
44 views

Does Googlebot respect the cache-control header?

Does Googlebot respect the cache-control header? By setting an appropriate cache-control header, can one improve the crawl efficiency?
0
votes
1answer
60 views

SEO Impact of Vary: User-Agent header

Vary: User-Agent header is used to tell browsers, CDNs, etc., (for caching purposes) that the content differs by user agents (especially for mobile and desktop versions). But do search engine robots ...
0
votes
0answers
27 views

Quickest way to remove many dynamic pages with duplicate content due to a spider trap

I have an angular JS site that is reliant on google rendering the pages. I created a spider trap and have about 9000 pages in the index which most are duplicate content. Each page that is currently ...
2
votes
1answer
98 views

Does Googlebot crawl JS or JSON inside an HTML page?

My Google Search tool is complaining about certain URLs it found on my site. These are intentionally unusable URLs; I'm not using them in any links. Think of them as unused meta-data. A) In one ...
0
votes
0answers
34 views

Block some subdomains using robots.txt -Angular website with IIS [duplicate]

We have a testing environment with the surprising sub-domain name tests and I would like to block it to search engines. How can I let Google scan my site https://example.com and block https://tests....
1
vote
1answer
49 views

How does authentication-required content get indexed/crawled?

Assume we have a VOD platform to watch online videos and it requires authentication to show that video or any kind of content. User has to be signed up and after that, they can access the videos etc. ...
1
vote
1answer
62 views

Why Googlebot is crawling only one page?

I am trying to understand why Google bot is crawling only one page only on my website. At the Google console I registered a small XML sitemap with 9 URLs. Only one URL I marked as a high priority (0.8)...
2
votes
2answers
61 views

Can I indicate (via robots.txt, meta robots or another approach) that one or more queryString parameters should be ignored by crawlers?

I've written my own SiteSearch Script in PHP. The SiteSearch parses two GET parameters from the queryString: search // the search-phrase filters (optional) // indicating which parts of the site to ...
0
votes
0answers
31 views

In my Crawl Stats, 23% of requests are "Unauthorized" - is that bad?

Am I correct in thinking that having 23% of all crawl requests being blocked by "Unauthorized" 401/407 is NOT good? 68% of my crawl requests return 200 (OK) which is good of course. Is this ...
1
vote
1answer
91 views

Can search engine fetch/index website that using ipv6 only?

I see some host providers offer cheap server but only support ipv6. I want to rent it, but I am still considering the SEO impact. Can google and other search engine crawl and index it? If yes, will it ...
0
votes
0answers
38 views

How to block google from indexing mail record. duplication issue mail.example.com which is duplication of example.com

google is causing duplication by indexing mail.example.com which is identical to home page of www.example.com/ I added property in search console and removed this subdomain but keeps on indexing ...
2
votes
1answer
85 views

Do I need to submit sitemap.xml to google if I already have it in robots.txt and hosted on S3?

I have a static site hosted on S3 bucket which will have a robots.txt at root directory level. This robots.txt file will have reference to my sitemap.xml file which is hosted on another S3 bucket. It ...
2
votes
1answer
42 views

robots meta tag for specific web crawler

Web crawler list What I want noimageindex (Indexation-controlling parameters) will only work for googlebot (2 : two meta tag - code below) Others all crawlers (example as a YandexBot, Baiduspider) ...
2
votes
1answer
46 views

What is a difference in the order of "follow" and "index" in a robots meta tag?

What is the difference between <meta name="robots" content="follow, index"> and <meta name="robots" content="index, follow"> Would either create ...
2
votes
0answers
72 views

Prevent search engines from crawling and indexing all domains and subdomains set up under an Apache server

My company is renting a CentOS server, with one main domain and ~30 subdomains (the number varies, but it's usually around 30) hosted on it. The subdomains themselves are used for various purposes (...
5
votes
1answer
57 views

Consequences of blocking Semrush and other bots?

For typical websites, are there any disadvantages to my clients to blocking spiders like Semrush, Maxmind and the plethora of other "non search-engine" bots? Wouldn't blocking these ...
0
votes
0answers
34 views

What is most likely causing this index:1 404 error in Wordpress?

I'm running the latest version of WordPress on a VPS, and I recently moved my WordPress directory to a new location and reconfigured my site's permalink structure. So when you go to HTTP://www....
0
votes
0answers
15 views

Remove Google suggestions and links from search results [duplicate]

How to remove / delete google suggestions and related keywords links. As some results showing my personal data into some sites , need to remove those pages or links from search results showed by ...
1
vote
2answers
164 views

How to make a bot in Windows that does automated search of domain names to see if they are available? [closed]

Instead of manually typing in domain names in a domain search registry to see if they are available, How to program a bot to run through a pre-specified list of .com domain names by searching each one ...
2
votes
1answer
76 views

Is it legal to use the APIs of Meetup.com, Eventbrite etc, to aggregate their event data into one place, similar to Google?

I have a predicament. I'm trying to build a platform that will basically be a place where all events, online/offline, are listed. Initially - my platform will obviously have zero events, because ...
2
votes
0answers
29 views

Could allowing our images to be indexed hurt our web search rankings?

We are trying to solve some issues related to our SEO. When we compare ourselves to our competitors who are higher that us in SERPS, other than more backlinks (not much more), we don't see any ...
2
votes
0answers
110 views

Restrict Google from indexing subdomain from Azure CDN services

How can I stop Google from indexing example.azureedge.net content? We have the website from last two years but just recently we found out that Google has started to index all the URLs on the azure CDN ...
5
votes
2answers
132 views

How to keep crawler robots from indexing my page when there is a query string present in the URL?

I have a /products page. In that page, users may use filters to select the products they want to see. Those filters will work using query strings like: /products?brands=X,Y&size=1,2 and so on. I ...
0
votes
1answer
77 views

HTTP "Vary: User-Agent" header for a Single Page React App that does SSR for robots?

I have React web app. It's a single page app, so there's basically an "empty" HTML file with a JS bundle that does it all. But to improve SEO capabilities, I'm doing SSR for robot crawlers ...

1
2 3 4 5
17