Stack Exchange Network

Stack Exchange network consists of 175 Q&A communities including Stack Overflow, the largest, most trusted online community for developers to learn, share their knowledge, and build their careers.

Visit Stack Exchange
The results are in! See what nearly 90,000 developers picked as their most loved, dreaded, and desired coding languages and more in the 2019 Developer Survey.

Questions tagged [web-crawlers]

A computer program that accesses web pages for various purposes (to scrape content, to provide search engines with information about your site, etc.)

1
vote
1answer
11 views

Do I need to disallow mediapartners-google in robots.txt for SEO?

I found that most of the websites disallow mediapartners-google in robots.txt. Why do I need to do block mediapartners-google? Can blocking it help withp better SEO or does it not matter for SEO?
0
votes
0answers
6 views

SEO consequences for using IIS Rewrite rule to redirect to static content for crawlers only

I have an AngularJS site that's pretty slow which is causing the SEO to be rather poor. Not to mention, the metadata doesn't render correctly. To fix this, my thought was to use IIS rewrite rules to ...
0
votes
0answers
7 views

MediaWiki: Hiding drafts from Search engines without changing any file in the website directory

MediaWiki: Hiding drafts from Search engines without changing any file in the website directory, Is this possible or I must change robots.txt for drafts starting as draft:myDraft?
0
votes
1answer
21 views

Visitors using multiple IPs and determining if they are human

I'm seeing some obvious bot/script traffic that is bloating our Google Analytics(GA) data. I figure the traffic is using something that loads JS, CSS, etc. which is why it's affecting GA. So I put in ...
2
votes
0answers
23 views

Should a page with adsense be viewed at least once for adsense crawler to crawl?

I use plugins to insert Adsense codes on my WordPress website. The plugin has an option to disable ads for the admin user. Hence, I will not be seeing ads when creating and publishing/reviewing posts. ...
2
votes
2answers
28 views

Thousands of bot hits for retired domain

In May of last year we retired a domain and redirected it at a new one. This was done to consolidate two sites from a company merger into one site. We shutdown the CMS for site1.com and pointed its ...
1
vote
1answer
16 views

How do I access an entire website list that is continually generated when scrolling? [closed]

I am trying to acquire all the information from a list of songs on ranker, however as I keep scrolling down new songs appear in an ever growing list, as the name suggests its a voting website so I am ...
12
votes
1answer
19k views

Why does baidu crawl my site like crazy

When i'm checking my apache log I can see that baidu has been crawling my website 10 times a day for the last 2 weeks. It's not that i care much about it but I'm really curious about why he's doing ...
2
votes
1answer
377 views

Google intermittently fails to download CSS and JS from CloudFront CDN

We noticed a lot of "Mobile Usability" errors in the Search Console. When I clicked on them and chose "Check live version" option - it always said the page was mobile friendly. However, the mobile ...
4
votes
1answer
41 views

Robots.txt allow directory root while disallowing all deeper URLs

I would like to allow a path and disallow everything else in the directory. User-Agent: * Allow: /example/ Disallow: /example/* Will this work? If not what would be a better way of doing this?
0
votes
0answers
33 views

Website wrongly categorized as “Adult Content”

I have in the last few months I have placed my website content for micro blogging like zapier, tumblr,etc and pitching the same on many such sites. However few of these sites have not allowed the ...
0
votes
2answers
26 views

What happens if we forgot to write meta tags?

If we forgot to write meta tags, especially meta title and meta description, how will search engines crawl and index?
3
votes
9answers
12k views

Is this Google proxy a fake crawler: google-proxy-66-249-81-131.google.com?

Recently I discover that some variants of a google proxy visits my sites. I doubt these are legal Google crawlers because these crawlers are NOT always behind a proxy (like the hostname describes) and ...
2
votes
1answer
447 views

Fixing website title on Bing

I have recently created my own portfolio website, and the title meta tags specify the title of my front page as: "Thomas Russell's Portfolio and Blog", but when I search for "Thomas Russell's ...
0
votes
0answers
21 views

Google is marking as duplicate my unique URLs powered by Angular Universal

I have a problem in the search console, and google is marking my unique urls as duplicate The application is build with Angular Universal and is rendered in the server Note: This site is only ...
3
votes
2answers
381 views

Google Search Console: 404 errors on existing pages

There is a few years old small website with very few pages (~5), which were indexed and ranked by Google. A few days ago 4 of those pages disappeared from SERPs and Search Console gives me 404 Not ...
2
votes
1answer
43 views

Is blocking googlebot from accessing portions of the site with robots.txt penalized by Google?

I was reading this question. Let's say i have a div with something useful for users but i don't want it to be considered by Google. If i put this on an iframe and block the googlebot with robots.txt ...
1
vote
1answer
42 views

Clients from Microsoft IP address ranges seem to be spoofing Bingbot

I routinely get hits from Microsoft-owned IP addresses in the 13.x.y.z ranges of AS8075 (in these ranges) that present as Bingbot in the User Agent, but these addresses are listed on several block ...
6
votes
1answer
145 views

AdSense crawler access fails because the bot adds a slash to the login URL

I'm trying to get a site verified for AdSense. Since I want to display ads in authenticated pages too, I'm setting up Crawler Access to give a username and password to Google crawler. In this site, ...
1
vote
2answers
30 views

How to preserve ALL website's URLS when transferring to a new CMS as well hosting

I am in a position where I have to figure out how to preserve all of the website's pages URLS when transferring it to a new CMS, perhaps potentially Joomla (hasn't been decided yet). New hosting is ...
0
votes
2answers
45 views

Google keeps crawling old site, ignores new (& better) one

Searching 'Dragon Notes' in Google predominantly yields (1) rather than (2) - even for images; (2)'s almost non-existent. Differences: (2) is more up-to-date, has a better sitemap and source code (2) ...
1
vote
0answers
17 views

How to deal with great number of URLs that will not be in anchor and should be known only for friendly crawlers?

I have a hypothetical page, that contains pages like: example.com/see/cat example.com/see/mouse example.com/see/<basically anything> I can have thousands of such pages. The number of pages ...
4
votes
5answers
3k views

Google Bots Crawling my Site daily is this good or a bad thing?

Currently my site is being crawled by the Googlebot on a daily basis. Is a daily crawl from the Googlebot a good thing or bad thing?
2
votes
1answer
78 views

Two pages have same title, same description, different content. Are they duplicates?

We have release our website from our mobile app in a very short amount of time. The website is quite good, but not at all optimized for SEO. For example, there is absolutely no care for meta tags. All ...
1
vote
0answers
23 views

When using an image server on a separate domain, should we link directly to the image URL and restrict crawling, or redirect through our domain?

I have a large number of images stored on a separate server, these URLs currently go through a 302 redirect. I think these additional pages are a total waste of time for Google to crawl. At the ...
5
votes
1answer
217 views

Ok to use meta noindex on 404 pages?

I have a custom 404.html file on my website that gets served with a 404 status code when a URL is not found, standard stuff. But if you visit the exact path /404.html on the server, the server says "...
1
vote
0answers
33 views

changed site from http to https and now its unreachable by google [closed]

I move my website from http to https and now it does not appear in the google search resutls, it was appearing on the top search resutls. In google search consolte when I do fetch for the website it ...
0
votes
0answers
12 views

Only one page got indexed in the submitted sitemap [duplicate]

Google indexed only one URL in the sitemap I submitted. Please let me know if there's anything wrong with my sitemap. Any help appreciated. Example sitemap.xml <!-- language: xml --> <?xml ...
1
vote
1answer
31 views

Prevent dynamically generated thumbnails from being crawled

I have been using Raven Tools, and I want these dynamically generated thumbnails to be not crawled or not be flagged as No TITLE images. I tried adding X-Robots-Tag "noindex, nofollow" on each image ...
4
votes
3answers
97 views

How to prevent a competitor from cloning my website

I run a website that provides meta-data from specialized electronics (data like weight, dimensions, applications, voltage, etc). It also compare products using all this data so user can see the ...
1
vote
0answers
14 views

301 redirects to a similar domain

My company has two similar but not identical sites. Similar page hierarchy and content but different brands. We want to shut one down and 301 the pages to what we feel is relevant on the better site. ...
2
votes
1answer
32 views

Detecing bots by visits per second

Certain visitors on my site request more than one webpage per second on average. For example, I see the following statistics: ip visits timestsamps visits per second 1.2.3.4 500 500 ...
1
vote
0answers
142 views

Is email obfuscation still helpful in 2019? [closed]

Historically, even simple tricks would prevent mailto: links from being harvested by web crawlers, but I'm curious to know if modern bots are too advanced to be beaten. Is the effort to obfuscate ...
3
votes
1answer
203 views

How does changing an HTTP referrer header help circumvent crawler blocking

I have been researching different ways a web crawler might be blacklisted/blocked from a web-server and how to potentially circumvent that. One of those ways is to change the referrer header on the ...
4
votes
1answer
113 views

Will forbidding “bad bot” user agents via .htaccess add a layer of protection against email address scraping beyond the existing captcha?

I was reviewing the security/protection measures for the websites I manage (one is for a small consulting firm, the other for a not-for-profit art association), and I came across the topic of bad-bot ...
1
vote
0answers
18 views

Facing a challenge with the “Fetch As Google” feature in my webmaster

I am facing a challenge with the "Fetch As Google" feature in my webmaster Steps to reproduce bug:- 1. Goto webmaster , click on "Crawl" -> "Fetch As Google" 2. Select "Crawl this URL and its direct ...
0
votes
0answers
12 views

Why would a bot net keep loading my site?

In blocking a lot of scrapping activity, I found a puzzling situation - that my site is accessed about 35,000 times per day by around 30,000 different IPs. I started blocking these IPs with a script, ...
0
votes
1answer
82 views

Error in search console!

the new search console is pointing out usability errors on mobile devices on wordpress upload pages, that is, pages where I can not change font size, viewport window and things like that, how do I ...
1
vote
1answer
147 views

Is bad for SEO if browser need to run synchronous JS to get content?

I'm developing a site using symfony + Vue. The problem is that this HTML with non-standard tags (components) gets into the browser. All that contained in the <div id="app"> element must ...
1
vote
1answer
44 views

Does Google Crawl Rate have an impact on Google's Image Bot

I have a relatively large website. Each page has several images. Google indexes my website very slowly, but it actively crawls my pictures. Does changing the Google Crawl Rate have an impact on Google'...
2
votes
1answer
68 views

Is it good for SEO to change an article's slug when changing its title?

I have the following URL view: example.com/article/{slug}-{id} If I change the title of the article, then the slug will also change so that there is no disagreement. Please note that the article ...
5
votes
1answer
1k views

Why do many websites block requests from common HTTP libraries by user-agent?

Writing spiders, I have noticed that many sites will return a 403 error if I hit them from popular HTTP software libraries, unless I manually override the default User-Agent header used by the library....
2
votes
2answers
37 views

How many levels of sub-pages do Google bots crawl to when fetching and indexing linked pages in Google Search Console?

Let's say I submit a sitemap page that contains all the links I want Google to crawl, but instead of submitting a sitemap, I fetch the page and linked pages and let Google index them. How many levels ...
1
vote
1answer
78 views
1
vote
1answer
32 views

Should we Put Coupon Pages into sitemap because they do have a faster expiry rate?

Is it a good practice to keep coupons/ voucher pages into a sitemap. because they do have a faster expiry rate. Case: We are updating our sitemap once in 24 hours so, will it be a good practice to ...
11
votes
2answers
4k views

Should I block bots from my site and why?

My logs are full of bot visitors, often from Eastern Europe and China. The bots are identified as Ahrefs, Seznam, LSSRocketCrawler, Yandex, Sogou and so on. Should I block these bots from my site and ...
1
vote
1answer
43 views

SPA website, would making an HTML-only version for SEO be worthy?

I made a Vue.js SPA, and since it could affect SEO (specially because a large portion of my client base is not using Google), I was thinking: why not providing an HTML-only version for it? Would ...
7
votes
5answers
12k views

How to properly (dis)allow the archive.org bot? Did things change, if so when?

I have a website that I mostly don't want to be indexed by search engines, but I do want to preserve it for eternity on archive.org. So my robots.txt starts with this: User-agent: * Disallow: / ...
27
votes
8answers
20k views

Preventing robots from crawling specific part of a page

As a webmaster in charge of a tiny site that has a forum, I regularly receive complains from users that both the internal search engine and that external searches (like when using Google) are totally ...
1
vote
1answer
38 views

Can Google crawl .TS (MPEG2-TS) video format?

We are using the tool mp4hls (https://www.bento4.com/developers/hls/) to prep the hls stream. Basically, I think it can conver any video format to .mp4 then it convert to .ts. However, Google only ...