Stack Exchange Network

Stack Exchange network consists of 174 Q&A communities including Stack Overflow, the largest, most trusted online community for developers to learn, share their knowledge, and build their careers.

Visit Stack Exchange

Questions tagged [web-crawlers]

A computer program that accesses web pages for various purposes (to scrape content, to provide search engines with information about your site, etc.)

0
votes
1answer
33 views

Should I block web crawlers with robots.txt while my site is under maintenance?

I've recently inherited a quite popular website (fan page for a band), but the old webmaster had to take it down because of GDPR issues. There was a lot of content available on the website. I plan to ...
0
votes
0answers
18 views

Google Deindexing all our Pages - Strange Issue [duplicate]

We're facing deindexing issues for so many pages since two weeks and have lost rankings and traffic. When we check the URLs with inspection tool in webmaster, the canonical URLs are declared as ...
2
votes
1answer
265 views

Google intermittently fails to download CSS and JS from CloudFront CDN

We noticed a lot of "Mobile Usability" errors in the Search Console. When I clicked on them and chose "Check live version" option - it always said the page was mobile friendly. However, the mobile ...
1
vote
1answer
444 views

Google Search Console Crawl 404 Error for some wp-content directories even when those directories contain files

Here is a picture of my Google Search Console: When I try to click at the url for example error 18: wp-content/uploads/2016/01/ Appear File not found. Of course inside the path (wp-content/uploads/...
4
votes
1answer
242 views

Dynamic content using .append() or .innerHTML() and SEO

I was looking for some clarification on something I am looking to implement, I would like to know how a Google Crawl will see this and if it would be detrimental to SEO. On my webpage I have an empty ...
2
votes
1answer
538 views

Can Angular project be indexed in Google without Angular Universal?

I'm working on an Ionic PWAprogressive web app (PWA) and forthere are reasons iI cannot use Angular Universal. Is there any way to index with success inget Google to successfully index the site ...
2
votes
1answer
114 views

Is this site cloaking to hide my backlink from spiders?

I wrote a guest post at the following URL: https://wparena.com/integrate-google-tag-manager-wordpress/ At the end, you can see my author box with a URL pointing back to my site. This link is a ...
2
votes
0answers
14 views

Prevent bad rankings on a Q and A site for questions without an answer by using noindex until answered [duplicate]

I have a website on which users can ask a question & also they can answer it. So, when a user asks a question, it creates a new web-page on the website & below that question lies it's answers ...
1
vote
0answers
32 views

Do any search engines exist which disobey rel=“nofollow” or “Disallow”? [closed]

Moving beyond the big players - Google, Yahoo, Bing, Ask, DDG (who all obey "nofollow"s), are there any search engines out there which deliberately disobey rel='nofollow' links and Disallow rules in ...
0
votes
0answers
10 views

Removing Paginated URLs from Google Search [duplicate]

I have a website running on Django, Python. I'm using django pagination so that users can click next & see further content. For example, my website is example.com, so when user requests new page ...
1
vote
1answer
30 views

Another url is indexed in google which has a link ending with rpid=5PO84Q269

My Website is indexed in google. When i was searching for the term in google for my website i found a url placing above my websites root URL which has same url as root URL but has a weird url ...
1
vote
0answers
65 views

Google showing Pages with `noindex` meta tag [duplicate]

I have a website running on Django, Python. I have 2 simple questions, First: A week ago I made few changes in the main urls of website. For example https://example.com/user/account/profile is ...
0
votes
0answers
10 views

How to make Google show my blog page which is on a different domain?

I have a website (eg. https://work.example.com ) which is being crawled by google. I have a blog for the same under https://blogs.example.com. When I search in google with keywords like "work example",...
2
votes
0answers
125 views

Does Googlebot cache JavaScript files between two pages if they expire in one year?

My question is about javascript websites. Suppose GoogleBot has crawled example.com/index.html?id=1 Does it request all the javascript files again when crawling example.com/index.html?id=2? index....
3
votes
2answers
219 views

Is the top entry and exit page in Webalizer reports the most popular page on my site?

According to Webalizer, the top entry and exit page to my website is "/blog/". This is where my Wordpress blog is situated. (There are static non-Wordpress pages on my site as well.) Does that mean ...
2
votes
0answers
70 views

Crawling a SalesForce site fully generated by JS with Screaming Frog only gets content on some pages

I'm trying to crawl a Salesforce-generated site. This site appears to be fully generated by Javascript (if you load the page with JS disabled only a white screen shows up). For instance, Salesforce's ...
3
votes
3answers
417 views

Blocking bad bots in .htacess - should I add a BLOCK list or ALLOW list?

I got my Siteground hosting account suspended yesterday due to the allowed CPU overage. Besides recommendation to disable or control the WP heartbeat API, I was also recommended to block bad bots and ...
0
votes
0answers
9 views

URL not crawled to Google webmaster [duplicate]

This is the website sitemap url - https://www.geeks.sydney/sitemap.xml. I have submitted this url to Google search console/webmaster about 6-8 months ago. Still now I can see that, 171 url are indexed ...
6
votes
1answer
110 views

AdSense crawler access fails because the bot adds a slash to the login URL

I'm trying to get a site verified for AdSense. Since I want to display ads in authenticated pages too, I'm setting up Crawler Access to give a username and password to Google crawler. In this site, ...
1
vote
2answers
174 views

Delivering a different version of website to Google robots (SSR)

I am working on an enterprise website project as an SEO manager. They were using Angular.js by now but I made them switch to SSR (server-side rendering) since Google robots were not able to render ...
2
votes
0answers
113 views

Which webmail providers spider links in emails? [closed]

Does anyone know which webmail providers spider links in emails? I know that Outlook.com does, because I made a canary token and emailed it to an Outlook email address and the token got crawled ...
2
votes
1answer
80 views

How to make Google display only Country-specific search results based on user's location

My question goes like this: I own a multinational company with the following websites www.abc.com/us/ www.abc.com/uk/ www.abc.com/apac/ I advertise my product (let's name it "device") on each of my ...
-1
votes
1answer
71 views

Maximum time to cache HTTPS URL after HTTP to HTTPS migration?

We have migrated our site from HTTP to HTTPS. How much time will the Google crawler take to cache all our HTTPS pages. We have more than 50K pages in our website.
0
votes
1answer
75 views

How to identify all search engine bots and crawlers?

I'm recently doing keyword advertising on Bing, which also powers AOL and Yahoo searches, to promote a product. And I'm using a third-party tracker to track the clicks of my advertisements. I found ...
1
vote
0answers
15 views

crawling issue in my website about two week [duplicate]

I almost did this change before 20-21 days but still Google is not crawling Site.
0
votes
1answer
144 views

No page information in search results | Magento 2.1.8

When I google my domain, I can find it to be first but it's saying "No information is available for this page" The website has meta information like this: <meta name="description" content="My ...
1
vote
1answer
36 views

SEO crawler is reporting an issues with a high number of external links, is this a problem?

Deep Crawl shows an issue of high external linking (68) for my website. I thought external linking is good for the website! Is this an issue that needs to be fixed to improve SEO, and if so, what is ...
1
vote
2answers
127 views

Would multiple images with the same alt text as the page name be keyword stuffing?

In a page I have three or four images that have same alt tag as of my page name. Could Google consider that to be keyword and penalize my site? Is it good practice to include page name in alt tag? ...
1
vote
4answers
235 views

Big discrepancy in number of known pages in Google Search Console and ScreamingFrog

In a technical SEO audit I used both Google Search Console and ScreamingFrog crawl to review the status errors, however there is big discrepancy in the number of pages that Google knows about, and the ...
1
vote
2answers
146 views

How often do search bots typically visit a website?

A few weeks ago, I completed a website and submitted its Sitemap to Google. To date, I have had no real problems, with most of the URLs now successfully indexed. When a search engine returns for ...
1
vote
3answers
270 views

SEO impact of onclick handlers in place of links

I have a question similar to the one posted here. I'm building a React application which has very strict SEO requirements, so I'm wondering if the following approach is an issue: For all of our ...
0
votes
2answers
67 views

How to say, “Google, index my web site only during specific times?”

Is it possible tell Google to only index my site between certain hours? If it is, how can I manage it?
6
votes
2answers
670 views

Facebook crawler with no user agent spamming our site in possible DoS attack

Crawlers registered to Facebook (ipv6 ending in :face:b00c::1) were slamming our site, seeing 10s of thousands of hits in just 20 minutes. We noticed they didn't have a user agent in the header and ...
0
votes
0answers
109 views

Is there a way to get Google search console crawl stats for larger than 90 days?

Google search console allows you to see how many pages Google is crawling per day on your site, but it only goes back 90 days. You can view your crawl stats here: https://www.google.com/webmasters/...
3
votes
2answers
424 views

What is `/&wd=test` URL that is being requested from my site, probably by bots

I'm seeing error logs on a website because something tried to access: example.com/&wd=test the HTTP_REFERER is http://www.baidu.com/s?wd=FQQ the error is from ASP.net: A potentially ...
3
votes
2answers
112 views

How to force Google and other bots to pick actual images and not thumbnails?

For example, if there is an online shopping websites with thousands of small thumbnails of products and when you click over them it shows actual retina/full HD image over a popup. In this scenario, ...
1
vote
2answers
117 views

Pagination and crawl depths

On a website with ~40 blogs, we have recently switched on pagination meanings blogs are on page 1-8. With google crawl being less likely to crawl over 3 clicks deep, will pages not immediately linked ...
2
votes
3answers
409 views

How can I prevent bad bots from clicking on my advertisers' ads?

I have a website where I create ads for clients. They are not AdWords or similar, they are custom ads I create. All ad links go through a count-clicker before being sent to the advertiser's URL (ex. ...
1
vote
1answer
52 views

Do soft 404 errors on wiki sites caused by pages not yet created cause SEO problems?

I host a couple of wiki based sites, so there is a lot of content at various stages of generation, and lots of links to pages that have yet to be created. Of course, these links are generating soft ...
1
vote
1answer
103 views

404 or 302 Redirect - what to use for a url which may be used in the future but not available at the moment

My site lists blogs like this example.com/?status=blog&id=number I only have 3 blogs now (id=1, id=2 and id=3). I checked my log files and saw that search bots are manipulating the URLs and ...
3
votes
1answer
75 views

Will a dynamic robots.txt file that disallows crawling based on the time of day hurt SEO?

We have a serious traffic issue on our site and we want to eliminate crawlers as part of the problem. If we disallow crawling using robots.txt only during some hours, will it hurt our SEO?
1
vote
1answer
222 views

Website Home Page URL Not Show On Google Search [duplicate]

If i search like site:www.paka.tv my main homepage URL not showing, but categories of my site pages showing on results. How to show my website URL (www.pakka.tv) on google search.
4
votes
1answer
1k views

Why am I getting bot hits from compute-1.amazonaws.com?

I have a WordPress website with AWS on use, namely the Cloudfront service, to serve CSS, images and JS from the cloud. Lately, I noticed a lot of hits from bots coming from : IP: 54.236.71.87 ...
1
vote
1answer
72 views

Will creating an app that user prefer to the website reduce search engine rankings due to lower usage of the site?

If all users use only iPhone or Android app instead of use the website, will I lose rank of my website on Google? I think when more users visit the website, the rank of website is better in search ...
1
vote
1answer
725 views

How to find out the referrer of Googlebot's crawling URL?

Googlebot crawls 100s of 404 URLs from my website. I want to know from where it gets those links? Is there anything like HTTP Referrer? In detail: Googlebot crawls example.com/no_such_files.php ...
1
vote
2answers
605 views

Robots.txt isn't preventing my site from being crawled

I'm having problem with robots.txt. I put the robots.txt file in the website main directory (and also in /var/www/html - to make it work on all the server) but robots still keep crawling my websites. ...
1
vote
1answer
24 views

Keep order status private from search engines?

I searched hard before posting this question. I apologize if it is a duplicate or if this is not the correct forum. We have a homegrown shopping cart website. We do not require shoppers to register ...
0
votes
0answers
19 views

Home page has a youtube video on it, unsure how to fix google's description

A website i'm working on has embedded on it's homepage a youtube video. For some reason, google is ignoring my meta description and instead it is showing as a description the following message: "If ...
2
votes
1answer
42 views

Can we have Google crawl but ignore our paginated category pages and prefer our individual post pages in the index?

We have a site with a massive content back-end (50.000+) with, lets say, posts. We want to make these posts available to users, so people searching for the content on Google will find one of these ...
0
votes
1answer
101 views

How frequently should I expect Google AdsBot to visit my site?

I bought paid advertising for my website through an internet advertising company. I noticed that, among all the visits coming from that source, a large amount of them have AdsBot-Google (+http://www....