Stack Exchange Network

Stack Exchange network consists of 174 Q&A communities including Stack Overflow, the largest, most trusted online community for developers to learn, share their knowledge, and build their careers.

Visit Stack Exchange

Questions tagged [web-crawlers]

A computer program that accesses web pages for various purposes (to scrape content, to provide search engines with information about your site, etc.)

1
vote
2answers
26 views

How to preserve ALL website's URLS when transferring to a new CMS as well hosting

I am in a position where I have to figure out how to preserve all of the website's pages URLS when transferring it to a new CMS, perhaps potentially Joomla (hasn't been decided yet). New hosting is ...
0
votes
2answers
35 views

Google keeps crawling old site, ignores new (& better) one

Searching 'Dragon Notes' in Google predominantly yields (1) rather than (2) - even for images; (2)'s almost non-existent. Differences: (2) is more up-to-date, has a better sitemap and source code (2) ...
1
vote
0answers
14 views

How to deal with great number of URLs that will not be in anchor and should be known only for friendly crawlers?

I have a hypothetical page, that contains pages like: example.com/see/cat example.com/see/mouse example.com/see/<basically anything> I can have thousands of such pages. The number of pages ...
2
votes
1answer
41 views

Two pages have same title, same description, different content. Are they duplicates?

We have release our website from our mobile app in a very short amount of time. The website is quite good, but not at all optimized for SEO. For example, there is absolutely no care for meta tags. All ...
1
vote
0answers
16 views

When using an image server on a separate domain, should we link directly to the image URL and restrict crawling, or redirect through our domain?

I have a large number of images stored on a separate server, these URLs currently go through a 302 redirect. I think these additional pages are a total waste of time for Google to crawl. At the ...
3
votes
1answer
91 views

Ok to use meta noindex on 404 pages?

I have a custom 404.html file on my website that gets served with a 404 status code when a URL is not found, standard stuff. But if you visit the exact path /404.html on the server, the server says "...
1
vote
1answer
25 views

Clients from Microsoft IP address ranges seem to be spoofing Bingbot

I routinely get hits from Microsoft-owned IP addresses in the 13.x.y.z ranges of AS8075 (in these ranges) that present as Bingbot in the User Agent, but these addresses are listed on several block ...
1
vote
0answers
25 views

changed site from http to https and now its unreachable by google [closed]

I move my website from http to https and now it does not appear in the google search resutls, it was appearing on the top search resutls. In google search consolte when I do fetch for the website it ...
0
votes
0answers
16 views

Visitors using multiple IPs and determining if they are human

I'm seeing some obvious bot/script traffic that is bloating our Google Analytics(GA) data. I figure the traffic is using something that loads JS, CSS, etc. which is why it's affecting GA. So I put in ...
0
votes
0answers
12 views

Only one page got indexed in the submitted sitemap [duplicate]

Google indexed only one URL in the sitemap I submitted. Please let me know if there's anything wrong with my sitemap. Any help appreciated. Example sitemap.xml <!-- language: xml --> <?xml ...
1
vote
1answer
28 views

Prevent dynamically generated thumbnails from being crawled

I have been using Raven Tools, and I want these dynamically generated thumbnails to be not crawled or not be flagged as No TITLE images. I tried adding X-Robots-Tag "noindex, nofollow" on each image ...
4
votes
3answers
90 views

How to prevent a competitor from cloning my website

I run a website that provides meta-data from specialized electronics (data like weight, dimensions, applications, voltage, etc). It also compare products using all this data so user can see the ...
1
vote
0answers
12 views

301 redirects to a similar domain

My company has two similar but not identical sites. Similar page hierarchy and content but different brands. We want to shut one down and 301 the pages to what we feel is relevant on the better site. ...
2
votes
1answer
32 views

Detecing bots by visits per second

Certain visitors on my site request more than one webpage per second on average. For example, I see the following statistics: ip visits timestsamps visits per second 1.2.3.4 500 500 ...
1
vote
0answers
63 views

Is email obfuscation still helpful in 2019? [closed]

Historically, even simple tricks would prevent mailto: links from being harvested by web crawlers, but I'm curious to know if modern bots are too advanced to be beaten. Is the effort to obfuscate ...
3
votes
1answer
192 views

How does changing an HTTP referrer header help circumvent crawler blocking

I have been researching different ways a web crawler might be blacklisted/blocked from a web-server and how to potentially circumvent that. One of those ways is to change the referrer header on the ...
1
vote
0answers
15 views

Facing a challenge with the “Fetch As Google” feature in my webmaster

I am facing a challenge with the "Fetch As Google" feature in my webmaster Steps to reproduce bug:- 1. Goto webmaster , click on "Crawl" -> "Fetch As Google" 2. Select "Crawl this URL and its direct ...
0
votes
0answers
11 views

Why would a bot net keep loading my site?

In blocking a lot of scrapping activity, I found a puzzling situation - that my site is accessed about 35,000 times per day by around 30,000 different IPs. I started blocking these IPs with a script, ...
1
vote
1answer
141 views

Is bad for SEO if browser need to run synchronous JS to get content?

I'm developing a site using symfony + Vue. The problem is that this HTML with non-standard tags (components) gets into the browser. All that contained in the <div id="app"> element must ...
1
vote
1answer
32 views

Does Google Crawl Rate have an impact on Google's Image Bot

I have a relatively large website. Each page has several images. Google indexes my website very slowly, but it actively crawls my pictures. Does changing the Google Crawl Rate have an impact on Google'...
2
votes
1answer
57 views

Is it good for SEO to change an article's slug when changing its title?

I have the following URL view: example.com/article/{slug}-{id} If I change the title of the article, then the slug will also change so that there is no disagreement. Please note that the article ...
1
vote
1answer
48 views

How to add robots.txt to my WordPress site?

How to add robots.txt to my wordpress.com site?
1
vote
1answer
29 views

Should we Put Coupon Pages into sitemap because they do have a faster expiry rate?

Is it a good practice to keep coupons/ voucher pages into a sitemap. because they do have a faster expiry rate. Case: We are updating our sitemap once in 24 hours so, will it be a good practice to ...
2
votes
2answers
32 views

How many levels of sub-pages do Google bots crawl to when fetching and indexing linked pages in Google Search Console?

Let's say I submit a sitemap page that contains all the links I want Google to crawl, but instead of submitting a sitemap, I fetch the page and linked pages and let Google index them. How many levels ...
1
vote
1answer
32 views

Can Google crawl .TS (MPEG2-TS) video format?

We are using the tool mp4hls (https://www.bento4.com/developers/hls/) to prep the hls stream. Basically, I think it can conver any video format to .mp4 then it convert to .ts. However, Google only ...
1
vote
1answer
42 views

SPA website, would making an HTML-only version for SEO be worthy?

I made a Vue.js SPA, and since it could affect SEO (specially because a large portion of my client base is not using Google), I was thinking: why not providing an HTML-only version for it? Would ...
0
votes
1answer
60 views

Error in search console!

the new search console is pointing out usability errors on mobile devices on wordpress upload pages, that is, pages where I can not change font size, viewport window and things like that, how do I ...
1
vote
1answer
43 views

Google Bot Crawls using expired cookies and fails to get response

We are using a cookie on our website which is to fetch some information through an API. The cookie becomes invalid after a very small duration. Google Bot uses the same cookie again and again to ...
2
votes
1answer
125 views

htaccess 410 url spam hack requests excluding site generated request

Our site was hacked and links to random content where added to the site. We completely removed the hacked site and put a new one in its place. Everything new including images and content, no other ...
1
vote
2answers
291 views

Deleted tags & categories from my blog still come up in Google search results

I am a newbie in blogging. I've deleted some of the unwanted tags and categories from my wordpress blog but they kinda still come put in the search results when I search my website on Google. How do ...
4
votes
1answer
92 views

Will forbidding “bad bot” user agents via .htaccess add a layer of protection against email address scraping beyond the existing captcha?

I was reviewing the security/protection measures for the websites I manage (one is for a small consulting firm, the other for a not-for-profit art association), and I came across the topic of bad-bot ...
2
votes
0answers
36 views

Can many single page hits from a variety user agents from the same VPN with the same screen resolution be hurting my search rankings?

I am trying to clarify what is happening with the traffic on my site. Lately I am being flooded with many strange hits, mostly from US and they seem to only open 1 page or visit for few seconds. The ...
1
vote
1answer
20 views

SEO - Custom error pages & status codes

My website uses a custom error page with the <meta name="robots" content="noindex"> meta tag. Am I right in thinking this tag will tell crawlers, such as Googlebot, to remove the page from ...
1
vote
0answers
11 views

how do i search for certain word in a website? [closed]

Is there any way to search within a website without manually checking out every link? Suppose there is some website www.somewebsite.com and I want to search for a word in all of the pages www....
0
votes
2answers
49 views

Major Crawlers not honoring robots.txt?

few of my websites were compromised, there were thousands of links submitted to google, bing etc which are not relevant to my site. I cleaned the websites in question and updated the robots.txt. the ...
1
vote
1answer
25 views

Alternative languages redirecting

Serving a page in English, EG: https://www.example.com/en/make-games Has the following meta tags: <meta rel="alternate" href="https://www.example.com/fr/make-games" hreflang="fr" /> <meta ...
2
votes
2answers
130 views

Ampersand (&) in actual URL and sitemap

I have a sitemap (which is submitted to Google for indexing) in which I am adding URLs which have an ampersand (&) in it. Since in sitemap & has to be escaped I replaced & with &amp; ...
3
votes
1answer
98 views

How to hide HTML tags from SEO spiders

I understand and know how hiding things from your website works and how the search engine spiders will still be able to ¨see¨ this hidden element and penalize your website for it. Thing is, I did not ...
1
vote
1answer
182 views

Google started to reports root url blocked by robots.txt - but it's not -

My website has been running for multiple years without problem, however I recently noticed my root url on Google does not show any description/title. The Webmaster Tool reports this error: Crawl ...
3
votes
1answer
132 views

Should I publish an empty ads.txt file for a site that does not run any ads?

I have seen a few 404 errors when ads.txt is requested on some of the sites that I manage. I Guess the answer may be identical to the question about an empty robots.txt file or no file at all (just ...
5
votes
1answer
102 views

Does Google return to visit a page with 404 Error?

If a page indexed in Google shows a 404 Error, does Google return to visit it again? In other words, if a page that was indexed in Google continue to show the 404 Error for several years, will ...
0
votes
1answer
18 views

How deep do search engine crawlers go?

I have 2 sites with very similar HTML structure and I would like to know how google or any other crawler rank these 2 sites. Will they penalize me due the sites sharing a structure? The content ...
1
vote
1answer
67 views

Googlebot refuses to render more than index.html in Ember app

I am using Ember 3.0.0 to create a booking site. Title and meta tags are handled via ember-cli-meta-tags and seem to be rendered correctly when pages are opened in the web browser. However, Googlebot ...
0
votes
2answers
77 views

Stop Google from accsesing pages without robots.txt

Is it possible to get Google to stop crawling certain pages without using robots.txt? My understanding is robots.txt can block Google from accsesing a page, but if using noindex (meta or header) ...
1
vote
0answers
71 views

Stop getting traffic from Googlebot on high server load

In my website, we usually get a spike in the number of requests because of search engine crawlers (like Googlebot) harvesting the website for indexing. So what I thought of; is depending on the ...
1
vote
0answers
313 views

What is user agent ;__CT_JOB_ID__:?

I've been searching the web for an explanation of the sudden influx we've gotten of user agents of this format: ;__CT_JOB_ID__:68a72eee-353c-4b0b-82ae-0cb6c8ca3e6e; Each visitor has a unique IP. So ...
1
vote
0answers
26 views

Product carousel on my site is showing up blank in Google's fetch and render for mobile

Using GSC, I am trying to fetch a page with products and content, this page also has few products in carousel format and when I render this page for desktop or mobile, I am seeing everything fine but ...
2
votes
3answers
63 views

Is it better to start a website with a new domain or an existing domain that had downtime?

For the past year I have been working on a website online and offline. I would take the website down for weeks or months and work on it locally instead. Now that I am fully ready to launch it, is it ...
1
vote
0answers
13 views

Should a page with adsense be viewed at least once for adsense crawler to crawl?

I use plugins to insert Adsense codes on my WordPress website. The plugin has an option to disable ads for the admin user. Hence, I will not be seeing ads when creating and publishing/reviewing posts. ...
1
vote
2answers
47 views

MEAN stack app robots.txt not performing as expected

I am trying to use robots.txt to allow crawling public links on my SPA, but to disallow other links to the site. When I use the robots.txt tester by Google, it does not differentiate between the two ...