Stack Exchange Network

Stack Exchange network consists of 174 Q&A communities including Stack Overflow, the largest, most trusted online community for developers to learn, share their knowledge, and build their careers.

Visit Stack Exchange

Questions tagged [web-crawlers]

A computer program that accesses web pages for various purposes (to scrape content, to provide search engines with information about your site, etc.)

2
votes
1answer
442 views

htaccess empty referer deny “google bot”

I put this rule into htaccess file to deny empty referers which is returning 403 SetEnvIfNoCase Referer "^$" bad_user Deny from env=bad_user I can see the log, it's denying also Googlebot which ...
1
vote
0answers
21 views

confused with google url indexing [duplicate]

When enter my site in google like this site:mywebsite.com I have seen this below image. I have 169 total urls in my sitemap.xml file. It says 102 results found. But, when I go to next page (second ...
1
vote
1answer
84 views

How to stop indexing/crawling for Shop Checkout Summaries?

I have a small shop checkout that uses cookies for my cart and after the payment is done it generates a unique order-id creates an "overview" page with an url like this: http://localhost:8180/app_dev....
1
vote
0answers
33 views

How to prevent CDN content URLs being indexed by Google [duplicate]

Well, robots.txt prevent crawling and meta robots tag in HTML (or) X-Robots-Tag HTTP header prevents indexing (and other functionalities available too). So, even when a URL is disallowed in robots....
1
vote
1answer
24 views

Does Google or Bing follow dynamic HATEOS links that are not in an 'a' tag?

I have a Angular 1x SPA that uses the HATEOS standard to manage navigation. This is further complication by the links needing to be digested and processed to direct the user to the correct content ...
3
votes
1answer
78 views

What are these many requests referred from Facebook with a changing s= parameter?

I just recognised a peak in our access logs and I'm curious how to explain this or if my suggestion how to explain is right. There about 600 requests in 3 minutes with a referrer like this: http://l....
3
votes
1answer
228 views

How to prevent Googlebot from doing API requests?

I have a currency converter site around 32k pages. Every pair for each page. And every page has 2 API requests. I started to see huge number of API requests. From users I have only few page views ...
1
vote
2answers
468 views

How do you add a rule just for a specific bot to robots.txt?

I have a small website, for which the current robots.txt looks like so: User-agent: * Disallow: Sitemap: https://www.myawesomesite.com/sitemap.xml Now, I want to add another rule like so (with the ...
-1
votes
1answer
81 views

How to block everything from being indexed except sitemap.xml

I want to block everything and index sitemap.xml file alone. So I do it as shown below: User-agent: * Disallow: / Allow: /sitemap.xml Sitemap: https://example.net/sitemap.xml However, I am not sure ...
1
vote
0answers
9 views

Wordpress Website not indexing in google, no error on google webmaster [duplicate]

I just recognized my SEO Score is 0 and that my wordpress website is not indexed on google search engine, no automatic filling as well. I am not an expert, I never used a sitemap, nor htaccess but it ...
2
votes
0answers
17 views

When does a search engine learn about a particular user inside a website and display the profile link and /or a Login link along with search results? [duplicate]

How does Google search engine display user profiles or Login links along with a search results of some websites? because obviously the username or profile name is coming from the server side, so does ...
2
votes
1answer
198 views

Is the User-Agent “gce-spider” a well known scammer, a bad bot?

My website has been scammed using some "scamming-web-site steals my content through a proxy and serves the stolen content from their own domain and replaces my site in the SERP"-technique. The Apache ...
1
vote
1answer
289 views

Prevent Googlebot from crawling “access denied” error (403) of private forum that are reported in Google Search Console?

I'm running a website base it by vbulletin. Recently I moved one of my forums to a private section of my website so users and googlebot cannot access it there. Now I'm getting lots of access denied ...
2
votes
2answers
54 views

Moving a site from one subdomain to another subdomain, what to do with pages which aren't “mappable”?

In the process of moving a site from one subdomain to another subdomain. For a lot of the subfolders a 301 redirect to their respective pages on the new site. However what should I do with subdomains ...
1
vote
2answers
369 views

How can I stop Google from indexing “pretty links” external redirects from my WordPress site?

When I search site:[example].com in Google for my blog, the majority of the pages that are being indexed are "pretty links" to external sites. I created these links using the Pretty Links plugin so ...
1
vote
1answer
51 views

Can a user or a crawler see the source of a page that has been redirected via a 301?

Is it possible for a user or a web-crawler to see the contents/source code of a webdocument that is automatically redirected to another page via a 301 redirect? If it is possible? How could a user ...
0
votes
0answers
10 views

Preventing/blocking from crawling a specific user control of a page [duplicate]

Currently, google access/crawls a user control from the page given below- http://articles.mercola.com/sites/articles/archive/2017/07/20/do-fidget-spinners-help-anxiety-adhd.aspx Result of Google ...
1
vote
1answer
72 views

Repeated hits on my site from different IP addresses trying to access .aspx files using all my bandwidth [closed]

I checked my raw access files after being notified that my site has been limited over the last 24 hours and see different IP addresses trying to access files with .aspx extensions, see attached log ...
2
votes
0answers
179 views

What % can be regarded as “normal” “not viewed traffic” reported in AWStats?

Recently statistics for my site is about 50% "not viewed traffic". This seems much as many robots are blocked. What % of "not viewed traffic" can be regarded as "normal"? Is my site under attack when ...
2
votes
2answers
78 views

How can I avoid site search page duplicate title tag error in pagination of site search?

Google Search Console shows me my site search page duplicate Title Tag,
10
votes
1answer
3k views

Googlebot submitting thousands of requests to our map locater and using up the API quota

We have a store locater page on our customer's site. The end user enters their postcode and a search radius and we display results on a Google Map. Recently we had begun to notice that the site was ...
2
votes
0answers
163 views

Google is adding “Archive” to the title of tag and category pages and not using the meta description

I am wondering why Google is indexing my tags and categories as archives. Is that OK to be like that or I should change them? How to change them? Actually it looks so awkward! I have configured my ...
2
votes
1answer
92 views

Should I prevent search engines from indexing empty user profile pages? If so, how much content is enough for indexing?

I'm developing a social website for book readers, with public user profile pages. For each user, there are several pages available: The main user page (about me, last activity, ...) Several book ...
1
vote
2answers
89 views

Would an online queueing system for a website have an effect on google's ability to crawl the site?

If I was to implement a visitor queuing system for my website, where if visitor count is higher than n, place them in a queue. Would this have an effect on google's ability to crawl the site? Thanks
3
votes
3answers
105 views

How long before Google indexes a new (to me) domain that previously had spam and virus problems?

I registered a new ".com" domain, but when I added it to the Google Search Console I saw that there were problems with viruses and spam problems. I submitted a request for reconsideration. Google ...
1
vote
0answers
20 views

Is it bad for SEO to have a URL with nothing on it? [duplicate]

I have a WordPress site and for a very complicated reason I have to set some post type's template to empty file. Nothing. Which means I have a URL on my site (let's say: example.com/postype/the-best-...
1
vote
2answers
461 views

How to Crawl a website requires cookies for audit?

Situation: My Client's website requires cookies to access it. Users should choose (Language and country) to access the website. The problem is: Whenever I try to crawl the website using any software (...
0
votes
1answer
74 views

Can we find all backlinks to a webpage? [closed]

In Page and Brin's paper on PageRank, they say that while you are guaranteed to be able to find all links that point away from a page, the reverse is not true. My impression from reading the paper was ...
1
vote
1answer
34 views

If .htaccess is used to block my bot from accessing a particular directory, will I know this?

I'm working on a research project and I have a question. Say I would like to crawl all pages of a given site. In the case that my bot is blocked from accessing a certain portion of the site, I would ...
1
vote
0answers
187 views

Why would a bot submit a sign up form with fake info? [duplicate]

I have a sign up form where people can enter their name and address, then click on a submit button. However I am getting BOTs entering sign up details and submitting the form - or at least I presume ...
1
vote
1answer
37 views

How do we know, what is the source from which BING search indexed my webpage? - A Blog or website where my website link is placed

One of my web page with sensitive information was indexed by BING search and which in turn showing in the search results. Now that i solved by Robots meta tag in the header. However i would like to ...
3
votes
1answer
327 views

Where can I redirect exploit scanning bots?

I get a lot of those exploit scanning bots like the ones looking for a WordPress login (which I don't have) or guessing other exploitable URLs. Where is a sensible place to redirect them to via ....
1
vote
1answer
297 views

Facebot sometimes fails to parse open graph meta data, causing share failure

We maintain a WordPress-based website where Yoast plugin takes care of Open Graph meta tags generation. Recently we had a problem with likes disappearing after domain / protocol migration, but we have ...
2
votes
1answer
68 views

Why wouldn't a website apear in Google search results after robots.txt update to clean up hacked site?

I have a WordPress website which was hacked couple of days ago. I have tried to add up some security. By following couple of tutorials, I have updated my robots.txt file. But now my website is not ...
1
vote
1answer
29 views

Re-Indexing Home Page - HTML/DOM Change

(Site on WordPress) Currently, our company homepage is using images instead of divs to display our main products/solutions. As the SEO, I wanted to remove these images (which consist of mostly text) ...
5
votes
1answer
384 views

Are user agent names case-sensitive in robots.txt?

I'm blocking various bots in robots.txt and I was wondering if their names are case-sensitive. For example: User-agent: grapeshot Disallow: / Would it matter if I use grapeshot or GrapeShot?
3
votes
2answers
216 views

Could a custom crawler find unlisted web pages?

Example: A website has no sitemap.xml, no robots.txt, no index of those pages. Pages are not blocked, bots and humans have access, but they would need the URLs URL format: example.com/ofehdjtd/some-...
3
votes
1answer
330 views

Rule out third party scraping, but allow Google crawling

How to make scraping of own content through wget, httrack etc. impossible, but allow crawling through googlebot? This should be done without showing to googlebot other content, as to other user ...
0
votes
3answers
94 views

A top directory in a URL path returns a 404 error, will its subpages still be crawled?

I found a site in which one of the directories in the URL path returns 404 error, but the subpages are still live. Example URL: https://shop.edurite.com/blogs/news/summer-activities In this URL ...
0
votes
0answers
9 views

Google unable to Crawl and Fetch my Website [duplicate]

I created my website www.tribologyconference.com on April 09, 2017 and on the same day I submitted this url (with www and without) to google and added these properties in google search console. For 4-...
1
vote
0answers
32 views

How do spambots find and submit to email opt-in the end of a funnel?

I have an opt-in page that's being nuked right now with spambots. A simple webpage that I use to allow subscribers into my email list. The only way to get to that page was through one of my tracking ...
1
vote
1answer
55 views

How should I protect “secret” links I send in Emails from being indexed by search engines?

I have noticed that bing/msnbot tries to index pages that were only ever linked to in one single email. My guess is, that some kind of browser toolbar is responsible for this. Apart from robots.txt, ...
3
votes
1answer
24 views

Should article preview pages be crawled and indexed by search engines?

I have a page called "all articles" that loads previews of articles using AJAX. Because of that, the content won't be visible to search engines, so the page would be considered "poor content". Since ...
0
votes
1answer
181 views

Does adding CDN stops Google from crawling?

I want to add Cloudflare free CDN to my site but worried about whether Google bots will be able to crawl my site later on or not.
0
votes
1answer
4k views

How block only Yandex bot

Can you show me how will look robots.txt when I block only yandex bot, allow Google bot and block Yandex bot.
1
vote
2answers
309 views

How does robots.txt work with sites in subfolders?

I have a single web host with a number of other parked domains/sites in sub-directories, like this: example.com is the primary site and root directory of the web hosting. example.com/www.example2....
2
votes
2answers
203 views

Ajax Content from Blocked Resource

I have a site built in AngularJS. Most of the dynamically-loaded content comes from a Wordpress back-end that is separate from the AngularJS site. In fact, it's a completely different domain and we do ...
0
votes
0answers
28 views

SEO - Pages are blocked because google failed to get the resource since it is blocked by robots.txt [duplicate]

I would like to index all the pages in my angular site by Google. I used ngMeta in my application for dynamically apply page titles and description. Is the dynamic meta tags really a problem? When I ...
1
vote
1answer
28 views

How do you properly SEO-tag an app that is only a catalogue of images?

I am making a little helper app that mostly links a lot of graphs and images. There is also some text explaining a couple of things, but more importantly that text will probably never (or very rarely) ...
2
votes
3answers
77 views

Is it good to submit our site in website submission sites?

I was wondering why don't we submit our site in website submission sites and increase our site's rank. Is is good to do this? How is this working? Is this a spam activity? What are the advantages ...