Questions tagged [web-crawlers]

A computer program that accesses web pages for various purposes (to scrape content, to provide search engines with information about your site, etc.)

Filter by
Sorted by
Tagged with
1
vote
1answer
59 views

Do soft 404 errors on wiki sites caused by pages not yet created cause SEO problems?

I host a couple of wiki based sites, so there is a lot of content at various stages of generation, and lots of links to pages that have yet to be created. Of course, these links are generating soft ...
1
vote
1answer
157 views

404 or 302 Redirect - what to use for a url which may be used in the future but not available at the moment

My site lists blogs like this example.com/?status=blog&id=number I only have 3 blogs now (id=1, id=2 and id=3). I checked my log files and saw that search bots are manipulating the URLs and ...
3
votes
1answer
84 views

Will a dynamic robots.txt file that disallows crawling based on the time of day hurt SEO?

We have a serious traffic issue on our site and we want to eliminate crawlers as part of the problem. If we disallow crawling using robots.txt only during some hours, will it hurt our SEO?
1
vote
1answer
367 views

Website Home Page URL Not Show On Google Search [duplicate]

If i search like site:www.paka.tv my main homepage URL not showing, but categories of my site pages showing on results. How to show my website URL (www.pakka.tv) on google search.
4
votes
1answer
2k views

Why am I getting bot hits from compute-1.amazonaws.com?

I have a WordPress website with AWS on use, namely the Cloudfront service, to serve CSS, images and JS from the cloud. Lately, I noticed a lot of hits from bots coming from : IP: 54.236.71.87 ...
1
vote
1answer
74 views

Will creating an app that user prefer to the website reduce search engine rankings due to lower usage of the site?

If all users use only iPhone or Android app instead of use the website, will I lose rank of my website on Google? I think when more users visit the website, the rank of website is better in search ...
1
vote
1answer
1k views

How to find out the referrer of Googlebot's crawling URL?

Googlebot crawls 100s of 404 URLs from my website. I want to know from where it gets those links? Is there anything like HTTP Referrer? In detail: Googlebot crawls example.com/no_such_files.php ...
1
vote
2answers
1k views

Robots.txt isn't preventing my site from being crawled

I'm having problem with robots.txt. I put the robots.txt file in the website main directory (and also in /var/www/html - to make it work on all the server) but robots still keep crawling my websites. ...
1
vote
1answer
26 views

Keep order status private from search engines?

I searched hard before posting this question. I apologize if it is a duplicate or if this is not the correct forum. We have a homegrown shopping cart website. We do not require shoppers to register ...
2
votes
1answer
42 views

Can we have Google crawl but ignore our paginated category pages and prefer our individual post pages in the index?

We have a site with a massive content back-end (50.000+) with, lets say, posts. We want to make these posts available to users, so people searching for the content on Google will find one of these ...
0
votes
1answer
156 views

How frequently should I expect Google AdsBot to visit my site?

I bought paid advertising for my website through an internet advertising company. I noticed that, among all the visits coming from that source, a large amount of them have AdsBot-Google (+http://www....
2
votes
1answer
618 views

htaccess empty referer deny “google bot”

I put this rule into htaccess file to deny empty referers which is returning 403 SetEnvIfNoCase Referer "^$" bad_user Deny from env=bad_user I can see the log, it's denying also Googlebot which ...
1
vote
0answers
26 views

confused with google url indexing [duplicate]

When enter my site in google like this site:mywebsite.com I have seen this below image. I have 169 total urls in my sitemap.xml file. It says 102 results found. But, when I go to next page (second ...
1
vote
1answer
123 views

How to stop indexing/crawling for Shop Checkout Summaries?

I have a small shop checkout that uses cookies for my cart and after the payment is done it generates a unique order-id creates an "overview" page with an url like this: http://localhost:8180/app_dev....
1
vote
0answers
185 views

How to prevent CDN content URLs being indexed by Google [duplicate]

Well, robots.txt prevent crawling and meta robots tag in HTML (or) X-Robots-Tag HTTP header prevents indexing (and other functionalities available too). So, even when a URL is disallowed in robots....
1
vote
1answer
27 views

Does Google or Bing follow dynamic HATEOS links that are not in an 'a' tag?

I have a Angular 1x SPA that uses the HATEOS standard to manage navigation. This is further complication by the links needing to be digested and processed to direct the user to the correct content ...
3
votes
1answer
88 views

What are these many requests referred from Facebook with a changing s= parameter?

I just recognised a peak in our access logs and I'm curious how to explain this or if my suggestion how to explain is right. There about 600 requests in 3 minutes with a referrer like this: http://l....
3
votes
1answer
337 views

How to prevent Googlebot from doing API requests?

I have a currency converter site around 32k pages. Every pair for each page. And every page has 2 API requests. I started to see huge number of API requests. From users I have only few page views ...
1
vote
2answers
648 views

How do you add a rule just for a specific bot to robots.txt?

I have a small website, for which the current robots.txt looks like so: User-agent: * Disallow: Sitemap: https://www.myawesomesite.com/sitemap.xml Now, I want to add another rule like so (with the ...
-1
votes
1answer
192 views

How to block everything from being indexed except sitemap.xml

I want to block everything and index sitemap.xml file alone. So I do it as shown below: User-agent: * Disallow: / Allow: /sitemap.xml Sitemap: https://example.net/sitemap.xml However, I am not sure ...
1
vote
0answers
9 views

Wordpress Website not indexing in google, no error on google webmaster [duplicate]

I just recognized my SEO Score is 0 and that my wordpress website is not indexed on google search engine, no automatic filling as well. I am not an expert, I never used a sitemap, nor htaccess but it ...
2
votes
0answers
18 views

When does a search engine learn about a particular user inside a website and display the profile link and /or a Login link along with search results? [duplicate]

How does Google search engine display user profiles or Login links along with a search results of some websites? because obviously the username or profile name is coming from the server side, so does ...
2
votes
1answer
242 views

Is the User-Agent “gce-spider” a well known scammer, a bad bot?

My website has been scammed using some "scamming-web-site steals my content through a proxy and serves the stolen content from their own domain and replaces my site in the SERP"-technique. The Apache ...
1
vote
1answer
325 views

Prevent Googlebot from crawling “access denied” error (403) of private forum that are reported in Google Search Console?

I'm running a website base it by vbulletin. Recently I moved one of my forums to a private section of my website so users and googlebot cannot access it there. Now I'm getting lots of access denied ...
2
votes
2answers
59 views

Moving a site from one subdomain to another subdomain, what to do with pages which aren't “mappable”?

In the process of moving a site from one subdomain to another subdomain. For a lot of the subfolders a 301 redirect to their respective pages on the new site. However what should I do with subdomains ...
1
vote
2answers
556 views

How can I stop Google from indexing “pretty links” external redirects from my WordPress site?

When I search site:[example].com in Google for my blog, the majority of the pages that are being indexed are "pretty links" to external sites. I created these links using the Pretty Links plugin so ...
1
vote
1answer
52 views

Can a user or a crawler see the source of a page that has been redirected via a 301?

Is it possible for a user or a web-crawler to see the contents/source code of a webdocument that is automatically redirected to another page via a 301 redirect? If it is possible? How could a user ...
1
vote
1answer
82 views

Repeated hits on my site from different IP addresses trying to access .aspx files using all my bandwidth [closed]

I checked my raw access files after being notified that my site has been limited over the last 24 hours and see different IP addresses trying to access files with .aspx extensions, see attached log ...
2
votes
0answers
217 views

What % can be regarded as “normal” “not viewed traffic” reported in AWStats?

Recently statistics for my site is about 50% "not viewed traffic". This seems much as many robots are blocked. What % of "not viewed traffic" can be regarded as "normal"? Is my site under attack when ...
2
votes
2answers
82 views

How can I avoid site search page duplicate title tag error in pagination of site search?

Google Search Console shows me my site search page duplicate Title Tag,
10
votes
1answer
3k views

Googlebot submitting thousands of requests to our map locater and using up the API quota

We have a store locater page on our customer's site. The end user enters their postcode and a search radius and we display results on a Google Map. Recently we had begun to notice that the site was ...
2
votes
0answers
229 views

Google is adding “Archive” to the title of tag and category pages and not using the meta description

I am wondering why Google is indexing my tags and categories as archives. Is that OK to be like that or I should change them? How to change them? Actually it looks so awkward! I have configured my ...
2
votes
1answer
129 views

Should I prevent search engines from indexing empty user profile pages? If so, how much content is enough for indexing?

I'm developing a social website for book readers, with public user profile pages. For each user, there are several pages available: The main user page (about me, last activity, ...) Several book ...
1
vote
2answers
97 views

Would an online queueing system for a website have an effect on google's ability to crawl the site?

If I was to implement a visitor queuing system for my website, where if visitor count is higher than n, place them in a queue. Would this have an effect on google's ability to crawl the site? Thanks
3
votes
3answers
116 views

How long before Google indexes a new (to me) domain that previously had spam and virus problems?

I registered a new ".com" domain, but when I added it to the Google Search Console I saw that there were problems with viruses and spam problems. I submitted a request for reconsideration. Google ...
1
vote
0answers
20 views

Is it bad for SEO to have a URL with nothing on it? [duplicate]

I have a WordPress site and for a very complicated reason I have to set some post type's template to empty file. Nothing. Which means I have a URL on my site (let's say: example.com/postype/the-best-...
1
vote
2answers
676 views

How to Crawl a website requires cookies for audit?

Situation: My Client's website requires cookies to access it. Users should choose (Language and country) to access the website. The problem is: Whenever I try to crawl the website using any software (...
0
votes
1answer
101 views

Can we find all backlinks to a webpage? [closed]

In Page and Brin's paper on PageRank, they say that while you are guaranteed to be able to find all links that point away from a page, the reverse is not true. My impression from reading the paper was ...
1
vote
1answer
41 views

If .htaccess is used to block my bot from accessing a particular directory, will I know this?

I'm working on a research project and I have a question. Say I would like to crawl all pages of a given site. In the case that my bot is blocked from accessing a certain portion of the site, I would ...
1
vote
0answers
235 views

Why would a bot submit a sign up form with fake info? [duplicate]

I have a sign up form where people can enter their name and address, then click on a submit button. However I am getting BOTs entering sign up details and submitting the form - or at least I presume ...
1
vote
1answer
40 views

How do we know, what is the source from which BING search indexed my webpage? - A Blog or website where my website link is placed

One of my web page with sensitive information was indexed by BING search and which in turn showing in the search results. Now that i solved by Robots meta tag in the header. However i would like to ...
3
votes
1answer
501 views

Where can I redirect exploit scanning bots?

I get a lot of those exploit scanning bots like the ones looking for a WordPress login (which I don't have) or guessing other exploitable URLs. Where is a sensible place to redirect them to via ....
1
vote
1answer
379 views

Facebot sometimes fails to parse open graph meta data, causing share failure

We maintain a WordPress-based website where Yoast plugin takes care of Open Graph meta tags generation. Recently we had a problem with likes disappearing after domain / protocol migration, but we have ...
2
votes
1answer
71 views

Why wouldn't a website apear in Google search results after robots.txt update to clean up hacked site?

I have a WordPress website which was hacked couple of days ago. I have tried to add up some security. By following couple of tutorials, I have updated my robots.txt file. But now my website is not ...
1
vote
1answer
33 views

Re-Indexing Home Page - HTML/DOM Change

(Site on WordPress) Currently, our company homepage is using images instead of divs to display our main products/solutions. As the SEO, I wanted to remove these images (which consist of mostly text) ...
4
votes
1answer
601 views

Are user agent names case-sensitive in robots.txt?

I'm blocking various bots in robots.txt and I was wondering if their names are case-sensitive. For example: User-agent: grapeshot Disallow: / Would it matter if I use grapeshot or GrapeShot?
2
votes
2answers
462 views

Could a custom crawler find unlisted web pages?

Example: A website has no sitemap.xml, no robots.txt, no index of those pages. Pages are not blocked, bots and humans have access, but they would need the URLs URL format: example.com/ofehdjtd/some-...
3
votes
1answer
399 views

Rule out third party scraping, but allow Google crawling

How to make scraping of own content through wget, httrack etc. impossible, but allow crawling through googlebot? This should be done without showing to googlebot other content, as to other user ...
0
votes
3answers
110 views

A top directory in a URL path returns a 404 error, will its subpages still be crawled?

I found a site in which one of the directories in the URL path returns 404 error, but the subpages are still live. Example URL: https://shop.edurite.com/blogs/news/summer-activities In this URL ...
1
vote
0answers
35 views

How do spambots find and submit to email opt-in the end of a funnel?

I have an opt-in page that's being nuked right now with spambots. A simple webpage that I use to allow subscribers into my email list. The only way to get to that page was through one of my tracking ...