Questions tagged [web-crawlers]

A computer program that accesses web pages for various purposes (to scrape content, to provide search engines with information about your site, etc.)

Filter by
Sorted by
Tagged with
-1 votes
1 answer
500 views

Does googlebot crawl the variable name of a php [closed]

Does Googlebot crawl the dynamic variable name aside from its value? For example in my PHP I use the variable _company_name. In the front-end is is _company_name saved in bot crawl also?
user avatar
0 votes
0 answers
14 views

Bingbot issues and crawling my website root domain [duplicate]

I seem to have this issue with a few different website's but bing is unable to crawl (or fetch the pages). I have done a cURL via windows and it connected properly however every URL live inspection ...
user avatar
2 votes
1 answer
22 views

How to modify google selected canonical URL when the URL is not valid anymore

I am the owner of two domains, domain1.com and domain2.com, A canonical URL : domain2.com is selected by google for domain1.com while the url does not exist anymore, how to make Google re-select ...
user avatar
  • 21
2 votes
1 answer
29 views

Can we request google to crawl our website every single day to update its indexes and tags on our domain?

I m thinking of making a blog website and noticed that google search results appear as per Google's indexing upon the crawled website. And this web crawling by google happens like once in 2 weeks to a ...
user avatar
  • 21
2 votes
1 answer
134 views

Will redirecting to a mailto URL prevent spam bots from harvesting the email address?

On https://www.heartinternet.uk/blog/15-ways-to-hide-your-email-address/, under the sub heading Replacing with a PHP script, a method to hide email addresses is presented that I like very much. On the ...
user avatar
0 votes
1 answer
43 views

Do Webcrawlers go to multiple subpages with no internal links?

I run a small business, for discussion let's say I have 4 pages: Page A (Home) Page B Page C Page D. Important: NONE of these Pages have internal links to each other. Imagine just 4 tabs at the top ...
user avatar
1 vote
0 answers
16 views

Exposing content from client-side headless CMS API to web crawlers

This is something I've been struggling to wrap my head around for the past few days. Tried to keep the rambling to a minimum but concise questions are at the bottom. As I understand it, utilizing JS ...
user avatar
1 vote
1 answer
25 views

Only allow crawling of pages in sitemap

My idea is, to use robots.txt to disallow everyting and put all allowed pages into the sitemaps. User-agent: * Disallow: / Sitemap: https://www.example.com/sitemap.xml Is this working as expected?
user avatar
  • 11
0 votes
0 answers
22 views

Do Baidu and/or Yandex publish the current IP ranges used by their site indexing bots?

I need to identify bots operated by search engine operators to index sites. I've looked at an existing question on this matter and the answers given there are too simplistic. The User-Agent header is ...
user avatar
1 vote
1 answer
39 views

Google Indexing Tons of Non-Existent URLs

A client that I am working for has come to me with a problem. Their Google Adwords account has been disabled because there is a bunch of traffic to crazy non-existent pages on their website. Here are ...
user avatar
0 votes
1 answer
29 views

Ratio of real visitors to search engine views

I know it is highly website-dependent, but is there any statistics/study/report on the ratio of real visitors websites get and the visits made by the search engine crawler? Services like Alexa reports ...
user avatar
  • 442
8 votes
1 answer
698 views

Are 'robots.txt' rules "starts with" or "contains" rules (clarification on confusing Google documentation)

Searching for robots.txt pattern matching rules (inc. the non official Google rules), I found this Google dev page where it says the /fish rule matches any path that starts with /fish (...) it doesn'...
user avatar
  • 183
0 votes
0 answers
14 views

google crawling the link which is dynamically changed

I have a question about how Google crawls. I have a pagination link on the site. Like this. www.domeinnaam.com/category?page=2 I suppose that google by crawling is going to click on this link. So far ...
user avatar
0 votes
0 answers
11 views

HOW TO GET MUTIPLE PAGES INDEXED [duplicate]

I have a website with approximately 700k web pages but only has 2k indexed pages. I have tried increasing the crawl limit to 2 requests per second because that is what google search console allows for ...
user avatar
  • 1
1 vote
2 answers
121 views

Can a Web Crawler follow links from <Link> tag in React?

I am building a frontend website and I am trying to make sure that the website can be crawled by search engines so that it can appear in search results. Currently, most of my links look like this in ...
user avatar
2 votes
1 answer
44 views

How to use Subdomain for a custom CMS of the main domain website?

Should I use a subdomain for a custom CMS of the main domain website? Like cms.domain.com. As it will be a CMS, I will disallow a search crawler for indexing. Will this hurt SEO for the Main Domain?
user avatar
4 votes
1 answer
74 views

Is there a way to allow ad bots to crawl a website that has "noindex,nofollow" on it?

Due to duplicated content, there is a "noindex,nofollow" on the website, but in order to run ppc ads, there needs to be a way for the ads bots to crawl the website Can this be done without ...
user avatar
2 votes
2 answers
63 views

Should I exclude the assets folder from search engine crawlers in angular

I'm working with angular and had added the assets folder as element to exclude from the search engine crawlers, into my robots.txt file. But I now get the following error while testing the site health ...
user avatar
1 vote
0 answers
26 views

Disallow crawling of all links to a domain, except only specific URLs with parameters

I have a domain like example.com under which everything is blocked and links to an SPA. Like example.com, and we've disabled everything like so: User-agent: * Disallow: / Now this SPA uses parameters ...
user avatar
0 votes
0 answers
10 views

MY wordpress website cannot be indexed by google [duplicate]

I have been trying to index my newly created website on wordpress for some time. It was released in early December and only two pages have been indexed. Overall, the site has 9 pages and only two have ...
user avatar
  • 1
1 vote
0 answers
31 views

How can I let Google crawl my site for Programmable Search Engine (custom site search), but not have it show in public search results?

The answer for my question might be inherent in the question itself- that if I want Google to crawl my site for a custom site search, such as through Programmable Search Engine, that it is those same ...
user avatar
3 votes
2 answers
49 views

Should we skip extra scripts (trackers, CRM, ..) when page is crawled by a bot?

We can identify bots with their User-Agent (1, 2), but that's not so straightforward. The benefit is to skip large and probably useless resources for bots, and just serve core content What's the best ...
user avatar
  • 85
14 votes
8 answers
5k views

Is it possible to keep the existence of a domain secret?

If I bought a domain like a9j47fn83jd8j49.tld, and only a friend and I ever visit it, who would know about it? It sounds like for most TLDs, there is some reporting that has to happen after you buy a ...
user avatar
  • 243
2 votes
2 answers
321 views

Does submitting sitemap.xml once change the way the site's new pages will be crawled/discovered in the future?

My static site doesn't generate sitemap.xml automatically. Hundreds of pages have been modified recently (meta description, etc.). I would like Google to recrawl the whole website. Submitting the URLs ...
user avatar
  • 532
3 votes
1 answer
43 views

How do Google and Yahoo fetch content for pages behind a paywall or registration wall?

How do Google and Yahoo fetch content for pages that are behind a paywall or registration wall? I noticed that search results can show content that is not visible publicly without a paid or registered ...
user avatar
  • 33
3 votes
1 answer
141 views

Is it okay to have hyphens in brand/company name?

My domain name is without hyphens, but I promote/market my brand/company name with hyphens in between (example: 'company-name'). Is it okay to have such difference? Or will it affect in any way to SEO ...
user avatar
2 votes
2 answers
95 views

Will bots crawl my page generated by javascript

I use simple javascript to generate pages based on thousands of products stored in a JSON object. The page URLs will be like /productList?keyword=XXX&page=x. The keyowrd is what uses input through ...
user avatar
  • 485
11 votes
4 answers
6k views

Prevent Google from thinking my own bot is faking traffic on my website

I have a large website which I want to be visited by my own bot early on the morning, every day (for caching purpose). I want my bot visits 10,000 pages per day. My natural traffic is about 5000 pages ...
user avatar
5 votes
1 answer
611 views

Why does it cause issues to use robots.txt and a meta robots tag at the same time?

It seems common wisdom that using X-Robots-Tag/meta robots with robots.txt to block a URL from being indexed "can cause issues", e.g.: Using both X-Robots-Tag and meta robots on a URL is ...
user avatar
  • 153
4 votes
1 answer
235 views

Does google follow links discovered in JSON-LD?

Lets say I have a site with a page of 20 products which are fed through an API. The products are loaded using infinity scroll so the initial page render has 5 products tops. Upon scrolling it adds 5 ...
user avatar
  • 175
4 votes
2 answers
95 views

Is there any way I can defend against automated file download bots?

My company offers a very specifically tailored Android application to supplement our other software. Since users must have an account in our other software in order to use the Android application, is ...
user avatar
4 votes
1 answer
76 views

Weird bot requests in logs

I restored a website on a new server after it was down (mistakenly deleted) for a while, and it immediately got bombarded with requests to non-existing pages like /tissue-salts-for-weight-loss.html ...
user avatar
1 vote
1 answer
21 views

Robots.txt not blocking yet GSC URL Inspection Tool says it is

Recently I realized Disallow /was set on my staging sites. I didn't realize until I got a warning in Search Console... Mistakes happen I've removed the directive and cleared all caches at least 10 ...
user avatar
  • 4,743
4 votes
1 answer
190 views

Do SEO analysis bots have any advantage in terms of website traffic?

I have loads of traffic from SEO analysis bots (which are crawling for link analysis purpose only, rather than indexing for search engines) such as AhrefsBot/7.0; +http://ahrefs.com/robot/ SemrushBot/...
user avatar
  • 442
2 votes
1 answer
41 views

Are URLs with fragments or paths better for SEO?

Suppose I have a food blog (say www.myfoodblog.example) with several recipes (mango-icecream, banana-icecream, and berry-icecream). So what I understood for SEO is to have these keywords of recipes in ...
user avatar
  • 21
0 votes
1 answer
35 views

Can I tell who is accessing archive.org archived documents from my site from my server logs?

I currently host a website with a collection of some of my projects, where a few of them are currently undergoing a patent review process. I don't want these documents to be public (though they were ...
user avatar
1 vote
1 answer
61 views

Should I add links to the Sitemap that are disallowed in robots.txt?

My robots.txt contains the line Disallow: /search.php. Shall I nevertheless include http://example.com/search.php into my sitemap?
user avatar
  • 113
2 votes
1 answer
89 views

What is a robots.txt file and how can I use it

I want to use a robots.txt file for my website, which is currently displaying a 404 error when requesting it (e.g., https://www.example.com/robots.txt). How do I make one? There's no option to create ...
user avatar
1 vote
2 answers
83 views

How to include URLs generated from the WordPress API in my XML sitemap?

I have a website build in Angular 8, which among other things has a blog section. I have a sitemap.xml file that contains all the pages except the blog. The blog is created on WordPress so in order to ...
user avatar
2 votes
2 answers
180 views

What impact do text-based accordion elements have on SEO?

I'm looking for some consensus on what the current school of though is about accordion elements. Here is a page that uses them as an example: https://www.goaskeve.com/venues/goldsmiths-centre/ Will ...
user avatar
0 votes
1 answer
122 views

HTML sitemap with noindex, follow: Matt Cutts vs John Mueller

Ref: Do noindex,follow pages pass pagerank / link juice? I have a requirement wherein I want to create an HTML sitemap to help users navigate to deep pages. This sitemap page will be linked from ...
user avatar
  • 2,233
2 votes
1 answer
25 views

How do rule length and use of wildcards determine whether allow or disallow takes precedence in Robots.txt?

I refer to this webpage: https://developers.google.com/search/docs/advanced/robots/robots_txt Could someone elaborate on the difference between these two: http://example.com/page.htm allow: /page ...
user avatar
  • 41
2 votes
1 answer
53 views

SEO | Cloaking | Is it safe to remove Third-party / Analytical scripts for crawlers/bots? Is this considered as cloaking?

Is it considered as cloaking if I remove third party scripts such as google tag manager, other analytics based scripts tags for crawlers/bots? By removing these external javascripts I will surely have ...
user avatar
0 votes
2 answers
31 views

How to know when a new website comes into existence?

Is there a way to know when a new website comes into existence. What I mean is how crawlers such as search engines know if something exists in first place. So let's say a new website www.example.com ...
user avatar
1 vote
1 answer
86 views

Do I need to allow bots on /* if I disallow /specific-directory?

If I want all bots to not crawl /specific-directory, but I want them to operate "as usual" (thus: crawl) all other directories of my site, is this robots.txt entry adequate? # All Bots User-...
user avatar
1 vote
0 answers
85 views

Does Googlebot respect the cache-control header?

Does Googlebot respect the cache-control header? By setting an appropriate cache-control header, can one improve the crawl efficiency?
user avatar
  • 2,233
0 votes
1 answer
90 views

SEO Impact of Vary: User-Agent header

Vary: User-Agent header is used to tell browsers, CDNs, etc., (for caching purposes) that the content differs by user agents (especially for mobile and desktop versions). But do search engine robots ...
user avatar
  • 2,233
0 votes
0 answers
31 views

Quickest way to remove many dynamic pages with duplicate content due to a spider trap

I have an angular JS site that is reliant on google rendering the pages. I created a spider trap and have about 9000 pages in the index which most are duplicate content. Each page that is currently ...
user avatar
  • 41
2 votes
1 answer
340 views

Does Googlebot crawl JS or JSON inside an HTML page?

My Google Search tool is complaining about certain URLs it found on my site. These are intentionally unusable URLs; I'm not using them in any links. Think of them as unused meta-data. A) In one ...
user avatar
1 vote
1 answer
65 views

How does authentication-required content get indexed/crawled?

Assume we have a VOD platform to watch online videos and it requires authentication to show that video or any kind of content. User has to be signed up and after that, they can access the videos etc. ...
user avatar

1
2 3 4 5
18