Questions tagged [web-crawlers]

A computer program that accesses web pages for various purposes (to scrape content, to provide search engines with information about your site, etc.)

Filter by
Sorted by
Tagged with
1
vote
1answer
74 views

Google Bot Crawls using expired cookies and fails to get response

We are using a cookie on our website which is to fetch some information through an API. The cookie becomes invalid after a very small duration. Google Bot uses the same cookie again and again to ...
2
votes
1answer
359 views

htaccess 410 url spam hack requests excluding site generated request

Our site was hacked and links to random content where added to the site. We completely removed the hacked site and put a new one in its place. Everything new including images and content, no other ...
1
vote
2answers
305 views

Deleted tags & categories from my blog still come up in Google search results

I am a newbie in blogging. I've deleted some of the unwanted tags and categories from my wordpress blog but they kinda still come put in the search results when I search my website on Google. How do ...
4
votes
1answer
197 views

Will forbidding “bad bot” user agents via .htaccess add a layer of protection against email address scraping beyond the existing captcha?

I was reviewing the security/protection measures for the websites I manage (one is for a small consulting firm, the other for a not-for-profit art association), and I came across the topic of bad-bot ...
2
votes
0answers
42 views

Can many single page hits from a variety user agents from the same VPN with the same screen resolution be hurting my search rankings?

I am trying to clarify what is happening with the traffic on my site. Lately I am being flooded with many strange hits, mostly from US and they seem to only open 1 page or visit for few seconds. The ...
1
vote
1answer
22 views

SEO - Custom error pages & status codes

My website uses a custom error page with the <meta name="robots" content="noindex"> meta tag. Am I right in thinking this tag will tell crawlers, such as Googlebot, to remove the page from ...
1
vote
0answers
13 views

how do i search for certain word in a website? [closed]

Is there any way to search within a website without manually checking out every link? Suppose there is some website www.somewebsite.com and I want to search for a word in all of the pages www....
0
votes
2answers
98 views

Major Crawlers not honoring robots.txt?

few of my websites were compromised, there were thousands of links submitted to google, bing etc which are not relevant to my site. I cleaned the websites in question and updated the robots.txt. the ...
1
vote
1answer
31 views

Alternative languages redirecting

Serving a page in English, EG: https://www.example.com/en/make-games Has the following meta tags: <meta rel="alternate" href="https://www.example.com/fr/make-games" hreflang="fr" /> <meta ...
3
votes
2answers
1k views

Ampersand (&) in actual URL and sitemap

I have a sitemap (which is submitted to Google for indexing) in which I am adding URLs which have an ampersand (&) in it. Since in sitemap & has to be escaped I replaced & with &amp; ...
3
votes
1answer
162 views

How to hide HTML tags from SEO spiders

I understand and know how hiding things from your website works and how the search engine spiders will still be able to ¨see¨ this hidden element and penalize your website for it. Thing is, I did not ...
2
votes
1answer
721 views

Google started to reports root url blocked by robots.txt - but it's not -

My website has been running for multiple years without problem, however I recently noticed my root url on Google does not show any description/title. The Webmaster Tool reports this error: Crawl ...
8
votes
1answer
1k views

Should I publish an empty ads.txt file for a site that does not run any ads?

I have seen a few 404 errors when ads.txt is requested on some of the sites that I manage. I Guess the answer may be identical to the question about an empty robots.txt file or no file at all (just ...
5
votes
1answer
179 views

Does Google return to visit a page with 404 Error?

If a page indexed in Google shows a 404 Error, does Google return to visit it again? In other words, if a page that was indexed in Google continue to show the 404 Error for several years, will ...
0
votes
1answer
18 views

How deep do search engine crawlers go?

I have 2 sites with very similar HTML structure and I would like to know how google or any other crawler rank these 2 sites. Will they penalize me due the sites sharing a structure? The content ...
1
vote
1answer
94 views

Googlebot refuses to render more than index.html in Ember app

I am using Ember 3.0.0 to create a booking site. Title and meta tags are handled via ember-cli-meta-tags and seem to be rendered correctly when pages are opened in the web browser. However, Googlebot ...
0
votes
2answers
99 views

Stop Google from accsesing pages without robots.txt

Is it possible to get Google to stop crawling certain pages without using robots.txt? My understanding is robots.txt can block Google from accsesing a page, but if using noindex (meta or header) ...
1
vote
0answers
185 views

Stop getting traffic from Googlebot on high server load

In my website, we usually get a spike in the number of requests because of search engine crawlers (like Googlebot) harvesting the website for indexing. So what I thought of; is depending on the ...
2
votes
0answers
683 views

What is user agent ;__CT_JOB_ID__:?

I've been searching the web for an explanation of the sudden influx we've gotten of user agents of this format: ;__CT_JOB_ID__:68a72eee-353c-4b0b-82ae-0cb6c8ca3e6e; Each visitor has a unique IP. So ...
1
vote
0answers
35 views

Product carousel on my site is showing up blank in Google's fetch and render for mobile

Using GSC, I am trying to fetch a page with products and content, this page also has few products in carousel format and when I render this page for desktop or mobile, I am seeing everything fine but ...
2
votes
3answers
70 views

Is it better to start a website with a new domain or an existing domain that had downtime?

For the past year I have been working on a website online and offline. I would take the website down for weeks or months and work on it locally instead. Now that I am fully ready to launch it, is it ...
2
votes
0answers
43 views

Should a page with adsense be viewed at least once for adsense crawler to crawl?

I use plugins to insert Adsense codes on my WordPress website. The plugin has an option to disable ads for the admin user. Hence, I will not be seeing ads when creating and publishing/reviewing posts. ...
1
vote
2answers
58 views

MEAN stack app robots.txt not performing as expected

I am trying to use robots.txt to allow crawling public links on my SPA, but to disallow other links to the site. When I use the robots.txt tester by Google, it does not differentiate between the two ...
0
votes
1answer
45 views

Should I block web crawlers with robots.txt while my site is under maintenance?

I've recently inherited a quite popular website (fan page for a band), but the old webmaster had to take it down because of GDPR issues. There was a lot of content available on the website. I plan to ...
2
votes
1answer
560 views

Google intermittently fails to download CSS and JS from CloudFront CDN

We noticed a lot of "Mobile Usability" errors in the Search Console. When I clicked on them and chose "Check live version" option - it always said the page was mobile friendly. However, the mobile ...
1
vote
1answer
678 views

Google Search Console Crawl 404 Error for some wp-content directories even when those directories contain files

Here is a picture of my Google Search Console: When I try to click at the url for example error 18: wp-content/uploads/2016/01/ Appear File not found. Of course inside the path (wp-content/uploads/...
4
votes
1answer
488 views

Dynamic content using .append() or .innerHTML() and SEO

I was looking for some clarification on something I am looking to implement, I would like to know how a Google Crawl will see this and if it would be detrimental to SEO. On my webpage I have an empty ...
3
votes
2answers
2k views

Can Angular project be indexed in Google without Angular Universal?

I'm working on an Ionic PWAprogressive web app (PWA) and forthere are reasons iI cannot use Angular Universal. Is there any way to index with success inget Google to successfully index the site ...
2
votes
1answer
158 views

Is this site cloaking to hide my backlink from spiders?

I wrote a guest post at the following URL: https://wparena.com/integrate-google-tag-manager-wordpress/ At the end, you can see my author box with a URL pointing back to my site. This link is a ...
2
votes
0answers
15 views

Prevent bad rankings on a Q and A site for questions without an answer by using noindex until answered [duplicate]

I have a website on which users can ask a question & also they can answer it. So, when a user asks a question, it creates a new web-page on the website & below that question lies it's answers ...
1
vote
0answers
46 views

Do any search engines exist which disobey rel=“nofollow” or “Disallow”? [closed]

Moving beyond the big players - Google, Yahoo, Bing, Ask, DDG (who all obey "nofollow"s), are there any search engines out there which deliberately disobey rel='nofollow' links and Disallow rules in ...
0
votes
0answers
22 views

Removing Paginated URLs from Google Search [duplicate]

I have a website running on Django, Python. I'm using django pagination so that users can click next & see further content. For example, my website is example.com, so when user requests new page ...
1
vote
1answer
44 views

Another url is indexed in google which has a link ending with rpid=5PO84Q269

My Website is indexed in google. When i was searching for the term in google for my website i found a url placing above my websites root URL which has same url as root URL but has a weird url ...
2
votes
0answers
180 views

Does Googlebot cache JavaScript files between two pages if they expire in one year?

My question is about javascript websites. Suppose GoogleBot has crawled example.com/index.html?id=1 Does it request all the javascript files again when crawling example.com/index.html?id=2? index....
3
votes
2answers
220 views

Is the top entry and exit page in Webalizer reports the most popular page on my site?

According to Webalizer, the top entry and exit page to my website is "/blog/". This is where my Wordpress blog is situated. (There are static non-Wordpress pages on my site as well.) Does that mean ...
2
votes
0answers
134 views

Crawling a SalesForce site fully generated by JS with Screaming Frog only gets content on some pages

I'm trying to crawl a Salesforce-generated site. This site appears to be fully generated by Javascript (if you load the page with JS disabled only a white screen shows up). For instance, Salesforce's ...
3
votes
3answers
833 views

Blocking bad bots in .htacess - should I add a BLOCK list or ALLOW list?

I got my Siteground hosting account suspended yesterday due to the allowed CPU overage. Besides recommendation to disable or control the WP heartbeat API, I was also recommended to block bad bots and ...
6
votes
1answer
216 views

AdSense crawler access fails because the bot adds a slash to the login URL

I'm trying to get a site verified for AdSense. Since I want to display ads in authenticated pages too, I'm setting up Crawler Access to give a username and password to Google crawler. In this site, ...
5
votes
2answers
699 views

Delivering a different version of website to Google robots (SSR)

I am working on an enterprise website project as an SEO manager. They were using Angular.js by now but I made them switch to SSR (server-side rendering) since Google robots were not able to render ...
2
votes
0answers
145 views

Which webmail providers spider links in emails? [closed]

Does anyone know which webmail providers spider links in emails? I know that Outlook.com does, because I made a canary token and emailed it to an Outlook email address and the token got crawled ...
2
votes
1answer
204 views

How to make Google display only Country-specific search results based on user's location

My question goes like this: I own a multinational company with the following websites www.abc.com/us/ www.abc.com/uk/ www.abc.com/apac/ I advertise my product (let's name it "device") on each of my ...
-1
votes
1answer
71 views

Maximum time to cache HTTPS URL after HTTP to HTTPS migration?

We have migrated our site from HTTP to HTTPS. How much time will the Google crawler take to cache all our HTTPS pages. We have more than 50K pages in our website.
0
votes
1answer
296 views

How to identify all search engine bots and crawlers?

I'm recently doing keyword advertising on Bing, which also powers AOL and Yahoo searches, to promote a product. And I'm using a third-party tracker to track the clicks of my advertisements. I found ...
1
vote
0answers
18 views

crawling issue in my website about two week [duplicate]

I almost did this change before 20-21 days but still Google is not crawling Site.
0
votes
1answer
230 views

No page information in search results | Magento 2.1.8

When I google my domain, I can find it to be first but it's saying "No information is available for this page" The website has meta information like this: <meta name="description" content="My ...
1
vote
1answer
60 views

SEO crawler is reporting an issues with a high number of external links, is this a problem?

Deep Crawl shows an issue of high external linking (68) for my website. I thought external linking is good for the website! Is this an issue that needs to be fixed to improve SEO, and if so, what is ...
2
votes
2answers
410 views

Would multiple images with the same alt text as the page name be keyword stuffing?

In a page I have three or four images that have same alt tag as of my page name. Could Google consider that to be keyword and penalize my site? Is it good practice to include page name in alt tag? ...
1
vote
4answers
359 views

Big discrepancy in number of known pages in Google Search Console and ScreamingFrog

In a technical SEO audit I used both Google Search Console and ScreamingFrog crawl to review the status errors, however there is big discrepancy in the number of pages that Google knows about, and the ...
1
vote
2answers
372 views

How often do search bots typically visit a website?

A few weeks ago, I completed a website and submitted its Sitemap to Google. To date, I have had no real problems, with most of the URLs now successfully indexed. When a search engine returns for ...
1
vote
3answers
870 views

SEO impact of onclick handlers in place of links

I have a question similar to the one posted here. I'm building a React application which has very strict SEO requirements, so I'm wondering if the following approach is an issue: For all of our ...

1 2 3 4 5 16