Questions tagged [web-crawlers]

A computer program that accesses web pages for various purposes (to scrape content, to provide search engines with information about your site, etc.)

Filter by
Sorted by
Tagged with
0
votes
0answers
14 views

Removing Paginated URLs from Google Search [duplicate]

I have a website running on Django, Python. I'm using django pagination so that users can click next & see further content. For example, my website is example.com, so when user requests new page ...
1
vote
1answer
33 views

Another url is indexed in google which has a link ending with rpid=5PO84Q269

My Website is indexed in google. When i was searching for the term in google for my website i found a url placing above my websites root URL which has same url as root URL but has a weird url ...
2
votes
0answers
163 views

Does Googlebot cache JavaScript files between two pages if they expire in one year?

My question is about javascript websites. Suppose GoogleBot has crawled example.com/index.html?id=1 Does it request all the javascript files again when crawling example.com/index.html?id=2? index....
3
votes
2answers
220 views

Is the top entry and exit page in Webalizer reports the most popular page on my site?

According to Webalizer, the top entry and exit page to my website is "/blog/". This is where my Wordpress blog is situated. (There are static non-Wordpress pages on my site as well.) Does that mean ...
2
votes
0answers
121 views

Crawling a SalesForce site fully generated by JS with Screaming Frog only gets content on some pages

I'm trying to crawl a Salesforce-generated site. This site appears to be fully generated by Javascript (if you load the page with JS disabled only a white screen shows up). For instance, Salesforce's ...
3
votes
3answers
633 views

Blocking bad bots in .htacess - should I add a BLOCK list or ALLOW list?

I got my Siteground hosting account suspended yesterday due to the allowed CPU overage. Besides recommendation to disable or control the WP heartbeat API, I was also recommended to block bad bots and ...
6
votes
1answer
187 views

AdSense crawler access fails because the bot adds a slash to the login URL

I'm trying to get a site verified for AdSense. Since I want to display ads in authenticated pages too, I'm setting up Crawler Access to give a username and password to Google crawler. In this site, ...
2
votes
2answers
420 views

Delivering a different version of website to Google robots (SSR)

I am working on an enterprise website project as an SEO manager. They were using Angular.js by now but I made them switch to SSR (server-side rendering) since Google robots were not able to render ...
2
votes
0answers
134 views

Which webmail providers spider links in emails? [closed]

Does anyone know which webmail providers spider links in emails? I know that Outlook.com does, because I made a canary token and emailed it to an Outlook email address and the token got crawled ...
2
votes
1answer
102 views

How to make Google display only Country-specific search results based on user's location

My question goes like this: I own a multinational company with the following websites www.abc.com/us/ www.abc.com/uk/ www.abc.com/apac/ I advertise my product (let's name it "device") on each of my ...
-1
votes
1answer
71 views

Maximum time to cache HTTPS URL after HTTP to HTTPS migration?

We have migrated our site from HTTP to HTTPS. How much time will the Google crawler take to cache all our HTTPS pages. We have more than 50K pages in our website.
0
votes
1answer
204 views

How to identify all search engine bots and crawlers?

I'm recently doing keyword advertising on Bing, which also powers AOL and Yahoo searches, to promote a product. And I'm using a third-party tracker to track the clicks of my advertisements. I found ...
1
vote
0answers
18 views

crawling issue in my website about two week [duplicate]

I almost did this change before 20-21 days but still Google is not crawling Site.
0
votes
1answer
203 views

No page information in search results | Magento 2.1.8

When I google my domain, I can find it to be first but it's saying "No information is available for this page" The website has meta information like this: <meta name="description" content="My ...
1
vote
1answer
42 views

SEO crawler is reporting an issues with a high number of external links, is this a problem?

Deep Crawl shows an issue of high external linking (68) for my website. I thought external linking is good for the website! Is this an issue that needs to be fixed to improve SEO, and if so, what is ...
2
votes
2answers
267 views

Would multiple images with the same alt text as the page name be keyword stuffing?

In a page I have three or four images that have same alt tag as of my page name. Could Google consider that to be keyword and penalize my site? Is it good practice to include page name in alt tag? ...
1
vote
4answers
315 views

Big discrepancy in number of known pages in Google Search Console and ScreamingFrog

In a technical SEO audit I used both Google Search Console and ScreamingFrog crawl to review the status errors, however there is big discrepancy in the number of pages that Google knows about, and the ...
1
vote
2answers
247 views

How often do search bots typically visit a website?

A few weeks ago, I completed a website and submitted its Sitemap to Google. To date, I have had no real problems, with most of the URLs now successfully indexed. When a search engine returns for ...
1
vote
3answers
553 views

SEO impact of onclick handlers in place of links

I have a question similar to the one posted here. I'm building a React application which has very strict SEO requirements, so I'm wondering if the following approach is an issue: For all of our ...
0
votes
2answers
69 views

How to say, “Google, index my web site only during specific times?”

Is it possible tell Google to only index my site between certain hours? If it is, how can I manage it?
7
votes
2answers
1k views

Facebook crawler with no user agent spamming our site in possible DoS attack

Crawlers registered to Facebook (ipv6 ending in :face:b00c::1) were slamming our site, seeing 10s of thousands of hits in just 20 minutes. We noticed they didn't have a user agent in the header and ...
3
votes
2answers
457 views

What is `/&wd=test` URL that is being requested from my site, probably by bots

I'm seeing error logs on a website because something tried to access: example.com/&wd=test the HTTP_REFERER is http://www.baidu.com/s?wd=FQQ the error is from ASP.net: A potentially ...
3
votes
2answers
149 views

How to force Google and other bots to pick actual images and not thumbnails?

For example, if there is an online shopping websites with thousands of small thumbnails of products and when you click over them it shows actual retina/full HD image over a popup. In this scenario, ...
2
votes
2answers
186 views

Pagination and crawl depths

On a website with ~40 blogs, we have recently switched on pagination meanings blogs are on page 1-8. With google crawl being less likely to crawl over 3 clicks deep, will pages not immediately linked ...
3
votes
3answers
638 views

How can I prevent bad bots from clicking on my advertisers' ads?

I have a website where I create ads for clients. They are not AdWords or similar, they are custom ads I create. All ad links go through a count-clicker before being sent to the advertiser's URL (ex. ...
1
vote
1answer
59 views

Do soft 404 errors on wiki sites caused by pages not yet created cause SEO problems?

I host a couple of wiki based sites, so there is a lot of content at various stages of generation, and lots of links to pages that have yet to be created. Of course, these links are generating soft ...
1
vote
1answer
149 views

404 or 302 Redirect - what to use for a url which may be used in the future but not available at the moment

My site lists blogs like this example.com/?status=blog&id=number I only have 3 blogs now (id=1, id=2 and id=3). I checked my log files and saw that search bots are manipulating the URLs and ...
3
votes
1answer
83 views

Will a dynamic robots.txt file that disallows crawling based on the time of day hurt SEO?

We have a serious traffic issue on our site and we want to eliminate crawlers as part of the problem. If we disallow crawling using robots.txt only during some hours, will it hurt our SEO?
1
vote
1answer
330 views

Website Home Page URL Not Show On Google Search [duplicate]

If i search like site:www.paka.tv my main homepage URL not showing, but categories of my site pages showing on results. How to show my website URL (www.pakka.tv) on google search.
4
votes
1answer
2k views

Why am I getting bot hits from compute-1.amazonaws.com?

I have a WordPress website with AWS on use, namely the Cloudfront service, to serve CSS, images and JS from the cloud. Lately, I noticed a lot of hits from bots coming from : IP: 54.236.71.87 ...
1
vote
1answer
74 views

Will creating an app that user prefer to the website reduce search engine rankings due to lower usage of the site?

If all users use only iPhone or Android app instead of use the website, will I lose rank of my website on Google? I think when more users visit the website, the rank of website is better in search ...
1
vote
1answer
1k views

How to find out the referrer of Googlebot's crawling URL?

Googlebot crawls 100s of 404 URLs from my website. I want to know from where it gets those links? Is there anything like HTTP Referrer? In detail: Googlebot crawls example.com/no_such_files.php ...
1
vote
2answers
870 views

Robots.txt isn't preventing my site from being crawled

I'm having problem with robots.txt. I put the robots.txt file in the website main directory (and also in /var/www/html - to make it work on all the server) but robots still keep crawling my websites. ...
1
vote
1answer
26 views

Keep order status private from search engines?

I searched hard before posting this question. I apologize if it is a duplicate or if this is not the correct forum. We have a homegrown shopping cart website. We do not require shoppers to register ...
2
votes
1answer
42 views

Can we have Google crawl but ignore our paginated category pages and prefer our individual post pages in the index?

We have a site with a massive content back-end (50.000+) with, lets say, posts. We want to make these posts available to users, so people searching for the content on Google will find one of these ...
0
votes
1answer
142 views

How frequently should I expect Google AdsBot to visit my site?

I bought paid advertising for my website through an internet advertising company. I noticed that, among all the visits coming from that source, a large amount of them have AdsBot-Google (+http://www....
2
votes
1answer
533 views

htaccess empty referer deny “google bot”

I put this rule into htaccess file to deny empty referers which is returning 403 SetEnvIfNoCase Referer "^$" bad_user Deny from env=bad_user I can see the log, it's denying also Googlebot which ...
1
vote
0answers
25 views

confused with google url indexing [duplicate]

When enter my site in google like this site:mywebsite.com I have seen this below image. I have 169 total urls in my sitemap.xml file. It says 102 results found. But, when I go to next page (second ...
1
vote
1answer
107 views

How to stop indexing/crawling for Shop Checkout Summaries?

I have a small shop checkout that uses cookies for my cart and after the payment is done it generates a unique order-id creates an "overview" page with an url like this: http://localhost:8180/app_dev....
1
vote
0answers
122 views

How to prevent CDN content URLs being indexed by Google [duplicate]

Well, robots.txt prevent crawling and meta robots tag in HTML (or) X-Robots-Tag HTTP header prevents indexing (and other functionalities available too). So, even when a URL is disallowed in robots....
1
vote
1answer
27 views

Does Google or Bing follow dynamic HATEOS links that are not in an 'a' tag?

I have a Angular 1x SPA that uses the HATEOS standard to manage navigation. This is further complication by the links needing to be digested and processed to direct the user to the correct content ...
3
votes
1answer
82 views

What are these many requests referred from Facebook with a changing s= parameter?

I just recognised a peak in our access logs and I'm curious how to explain this or if my suggestion how to explain is right. There about 600 requests in 3 minutes with a referrer like this: http://l....
3
votes
1answer
306 views

How to prevent Googlebot from doing API requests?

I have a currency converter site around 32k pages. Every pair for each page. And every page has 2 API requests. I started to see huge number of API requests. From users I have only few page views ...
1
vote
2answers
605 views

How do you add a rule just for a specific bot to robots.txt?

I have a small website, for which the current robots.txt looks like so: User-agent: * Disallow: Sitemap: https://www.myawesomesite.com/sitemap.xml Now, I want to add another rule like so (with the ...
-1
votes
1answer
167 views

How to block everything from being indexed except sitemap.xml

I want to block everything and index sitemap.xml file alone. So I do it as shown below: User-agent: * Disallow: / Allow: /sitemap.xml Sitemap: https://example.net/sitemap.xml However, I am not sure ...
1
vote
0answers
9 views

Wordpress Website not indexing in google, no error on google webmaster [duplicate]

I just recognized my SEO Score is 0 and that my wordpress website is not indexed on google search engine, no automatic filling as well. I am not an expert, I never used a sitemap, nor htaccess but it ...
2
votes
0answers
18 views

When does a search engine learn about a particular user inside a website and display the profile link and /or a Login link along with search results? [duplicate]

How does Google search engine display user profiles or Login links along with a search results of some websites? because obviously the username or profile name is coming from the server side, so does ...
2
votes
1answer
236 views

Is the User-Agent “gce-spider” a well known scammer, a bad bot?

My website has been scammed using some "scamming-web-site steals my content through a proxy and serves the stolen content from their own domain and replaces my site in the SERP"-technique. The Apache ...
1
vote
1answer
313 views

Prevent Googlebot from crawling “access denied” error (403) of private forum that are reported in Google Search Console?

I'm running a website base it by vbulletin. Recently I moved one of my forums to a private section of my website so users and googlebot cannot access it there. Now I'm getting lots of access denied ...
2
votes
2answers
59 views

Moving a site from one subdomain to another subdomain, what to do with pages which aren't “mappable”?

In the process of moving a site from one subdomain to another subdomain. For a lot of the subfolders a 301 redirect to their respective pages on the new site. However what should I do with subdomains ...