All Questions
Tagged with crawling or web-crawlers
913 questions
2
votes
1
answer
236
views
What is a difference in the order of "follow" and "index" in a robots meta tag?
What is the difference between
<meta name="robots" content="follow, index">
and
<meta name="robots" content="index, follow">
Would either create ...
- 21
9
votes
2
answers
5k
views
Facebook crawler with no user agent spamming our site in possible DoS attack
Crawlers registered to Facebook (ipv6 ending in :face:b00c::1) were slamming our site, seeing 10s of thousands of hits in just 20 minutes. We noticed they didn't have a user agent in the header and ...
- 4,023
2
votes
0
answers
463
views
Restrict Google from indexing subdomain from Azure CDN services
How can I stop Google from indexing example.azureedge.net content? We have the website from last two years but just recently we found out that Google has started to index all the URLs on the azure CDN ...
1
vote
2
answers
885
views
Does Google index page URLs with languages parameters as a different page?
If I implement multilingual access to the resource through the parameter:
1) https://example.com/en/article-1
2) https://example.com/article-1?l=en
So, does Google index both of them as a single ...
3
votes
2
answers
3k
views
Preventing links with query parameters from being indexed by Google
I manage a mobile version of a desktop site. The mobile site has an mdot subdomain and is served from a different server. As part of recommended best practices by Google, I've included a link on the ...
- 170
0
votes
0
answers
349
views
Why Google Cache Showing 404 on my React Base Webpages?
My web page is client-side rendered with React.
On hitting the cache, it is showing 404 instead of a rendered page.
Example Screenshot
When I inspect the Page URL on Search Console, Its parsing and ...
2
votes
1
answer
236
views
Will web crawlers be able to crawl nested sidebar menu for mobile web
I have a simple question. One clicking on any of the breadcrumbs, an overlay appears as per design. For example, in the above image clicking on Physics will bring up all subjects to choose from and ...
- 121
1
vote
1
answer
448
views
I have 2.1 million pages, how to get crawled by googlebot more frequently? [duplicate]
I have a project which aggregates public information on companies, startups, founders etc. for a total of >2 million tracked entities worldwide.
Yes, every single page is high quality, manually ...
- 111
6
votes
2
answers
2k
views
Delivering a different version of website to Google robots (SSR)
I am working on an enterprise website project as an SEO manager.
They were using Angular.js by now but I made them switch to SSR (server-side rendering) since Google robots were not able to render the ...
- 443
2
votes
4
answers
462
views
Secure way to share unfinished web-page with a customer?
What are the options to share unfinished web-page with a customer without allowing third-party to see the work in progress when developer has control over the web-server? I could come up with ...
- 289
2
votes
1
answer
2k
views
Ahrefs reports "Fetching robots.txt took too long"
My robots:
User-agent: *
Disallow: /wp-admin
Disallow: /sistema
Disallow: /site
Disallow: /old
Sitemap: http://www.example.com.br/page-sitemap.xml
In the google test tool: https://www.google.com/...
- 121
3
votes
2
answers
863
views
Why is my meta description blank in SERPs?
My client's site, which runs on Nuxt and Vue, is not showing meta descriptions on Google SERPs. They used to and as of last week they don't. Any idea why meta description tags would be stripped from ...
- 540
2
votes
2
answers
83
views
Can I indicate (via robots.txt, meta robots or another approach) that one or more queryString parameters should be ignored by crawlers?
I've written my own SiteSearch Script in PHP.
The SiteSearch parses two GET parameters from the queryString:
search // the search-phrase
filters (optional) // indicating which parts of the site to ...
- 2,365
3
votes
1
answer
760
views
New hostile web crawler from China throttling server
My server has been crashes several times the last 24 hours by a new extremely aggressive hostile distributed web crawler.
It is hitting over 1000 pages per minute and seems to be crawling all content ...
- 171
1
vote
1
answer
1k
views
Google Search Console: URL is not on Google (page with redirect)
I have just created a personal website which I host as github page and recently hooked it up with a custom domain. (In this post I changed the actual URLs to webmasters-example.com and webmasters-...
- 113
4
votes
0
answers
257
views
Using display: none VS a SPA approach, when it comes to SEO and performance
Say I have a component like,
// first approach (SPA)
const Component = () => {
const [show, toggle] = React.useReducer(show => !show, false)
return (
<div>
{show && &...
- 43
2
votes
1
answer
231
views
Do I need to submit sitemap.xml to google if I already have it in robots.txt and hosted on S3?
I have a static site hosted on S3 bucket which will have a robots.txt at root directory level.
This robots.txt file will have reference to my sitemap.xml file which is hosted on another S3 bucket. It ...
- 23
7
votes
7
answers
4k
views
Will banning bots make a site harder to find on search engines?
I'm running Apache 2 and a large part of our page views come from bots. Most of those are legitimate ones such as Google and Bing.
I want to parse the logs and get accurate statistics as to how many ...
- 175
2
votes
1
answer
87
views
robots meta tag for specific web crawler
Web crawler list
What I want
noimageindex (Indexation-controlling parameters) will only work for googlebot (2 : two meta tag - code below)
Others all crawlers (example as a YandexBot, Baiduspider) ...
- 155
2
votes
1
answer
614
views
Can I access the old version of Google Search Console to see crawling and indexing?
I remember google webmaster console had a crawl and indexing, But in new version, I couldn't find crawling or indexing. Everything is different.
Can I access to old style of google console?
- 269
0
votes
1
answer
686
views
SEO Impact of Vary: User-Agent header
Vary: User-Agent header is used to tell browsers, CDNs, etc., (for caching purposes) that the content differs by user agents (especially for mobile and desktop versions). But do search engine robots ...
- 2,343
3
votes
2
answers
1k
views
Am I getting DDOS from crawlers?
My website is currently getting hit with spamming bots (Example: 66.249.73.*) that are causing high CPU usage. Is it common for Google/Bing to crawl a website 10/sec?
I have done reverse look ups on ...
- 131
3
votes
1
answer
344
views
Why are bots repeatedly crawling pages of my site that I never created?
In my logs, I noticed that a lot of requests were coming from different bots and web-crawlers, but what confused me was that they were visiting pages such as /fpss/track/73/ that definitely do not ...
- 165
1
vote
0
answers
56
views
Exposing content from client-side headless CMS API to web crawlers
This is something I've been struggling to wrap my head around for the past few days. Tried to keep the rambling to a minimum but concise questions are at the bottom.
As I understand it, utilizing JS ...
1
vote
1
answer
326
views
Can I tell who is accessing archive.org archived documents from my site from my server logs?
I currently host a website with a collection of some of my projects, where a few of them are currently undergoing a patent review process. I don't want these documents to be public (though they were ...
user121961
0
votes
0
answers
224
views
Bingbot issues and crawling my website root domain [duplicate]
I seem to have this issue with a few different website's but bing is unable to crawl (or fetch the pages).
I have done a cURL via windows and it connected properly however every URL live inspection ...
4
votes
3
answers
5k
views
SEO impact of onclick handlers in place of links
I have a question similar to the one posted here.
I'm building a React application which has very strict SEO requirements, so I'm wondering if the following approach is an issue:
For all of our ...
- 175
4
votes
3
answers
1k
views
How can I prevent bad bots from clicking on my advertisers' ads?
I have a website where I create ads for clients. They are not AdWords or similar, they are custom ads I create. All ad links go through a count-clicker before being sent to the advertiser's URL (ex. ...
- 41
2
votes
1
answer
154
views
SEO | Cloaking | Is it safe to remove Third-party / Analytical scripts for crawlers/bots? Is this considered as cloaking?
Is it considered as cloaking if I remove third party scripts such as google tag manager, other analytics based scripts tags for crawlers/bots?
By removing these external javascripts I will surely have ...
12
votes
6
answers
17k
views
What is the best way to exclude bots from view counts?
My website is counting visitor views on certain pages. I noticed that Google and other Bots are "clicking" to my site like crazy and some of the pages get unrealistic view counts (compared to those ...
- 123
16
votes
4
answers
3k
views
Prevent malicious bots from posting spam
I remember a site closed due to misuse and I wonder if bots have a part of it. If the bot is POSTing something to my site what are ways I can combat it? I was thinking of setting some cookies and ...
user1585
2
votes
1
answer
312
views
Is it legal to use the APIs of Meetup.com, Eventbrite etc, to aggregate their event data into one place, similar to Google?
I have a predicament. I'm trying to build a platform that will basically be a place where all events, online/offline, are listed.
Initially - my platform will obviously have zero events, because ...
- 121
2
votes
1
answer
2k
views
Two pages have same title, same description, different content. Are they duplicates?
We have release our website from our mobile app in a very short amount of time.
The website is quite good, but not at all optimized for SEO. For example, there is absolutely no care for meta tags. All ...
- 123
1
vote
1
answer
397
views
Should I add links to the Sitemap that are disallowed in robots.txt?
My robots.txt contains the line Disallow: /search.php.
Shall I nevertheless include http://example.com/search.php into my sitemap?
- 113
1
vote
1
answer
386
views
Blocking bad bots based on useragent in apache conf
I am proposing blocking bad bots using the following code in /etc/sites-enabled/site.conf and I have a few questions.
RewriteEngine On
RewriteCond %{HTTP_USER_AGENT} useragent1|useragent2|useragent3 [...
- 11
3
votes
3
answers
1k
views
Using robots.txt to block sessionID URLs
I've got a problem with Googlebot literally crawling millions of URLs that are the same, because ";jsessionid" keeps getting inserted into the URL (something I cannot change due to the work ...
- 531
3
votes
1
answer
1k
views
Checking site for meta refresh redirects
How do you check or crawl site (couple of urls) to explore existing meta refresh redirects?
Screaming Frog doesn't handle them - it indicates a page with meta refresh as 200 (it is partly true), even ...
- 10.2k
0
votes
1
answer
269
views
Google not indexing product pages of my website [duplicate]
After many months and many changes to sitemaps/pages trying to solve the problem, Google still is not indexing the product pages of this site (link removed).
Here's the sitemap i'm using (link removed)...
- 3
3
votes
2
answers
2k
views
Does e.preventDefault on link prevent the page from being crawled?
I have a couple of links in navigation on homepage that lead to new pages. When clicked on link, i want to prevent going on that page (e.preventDefault()), and instead load that page in popup with ...
- 66
1
vote
2
answers
2k
views
Robots.txt isn't preventing my site from being crawled
I'm having problem with robots.txt.
I put the robots.txt file in the website main directory (and also in /var/www/html - to make it work on all the server) but robots still keep crawling my websites.
...
- 11
2
votes
1
answer
28
views
How do rule length and use of wildcards determine whether allow or disallow takes precedence in Robots.txt?
I refer to this webpage: https://developers.google.com/search/docs/advanced/robots/robots_txt
Could someone elaborate on the difference between these two:
http://example.com/page.htm
allow: /page
...
- 41
0
votes
1
answer
318
views
HTTP "Vary: User-Agent" header for a Single Page React App that does SSR for robots?
I have React web app. It's a single page app, so there's basically an "empty" HTML file with a JS bundle that does it all.
But to improve SEO capabilities, I'm doing SSR for robot crawlers ...
- 988
10
votes
1
answer
3k
views
Googlebot submitting thousands of requests to our map locater and using up the API quota
We have a store locater page on our customer's site. The end user enters their postcode and a search radius and we display results on a Google Map.
Recently we had begun to notice that the site was ...
- 287
2
votes
1
answer
529
views
Remove large amounts of pages from Bing Index Explorer
I have a website that I took over in my business. We redid the whole website (which was previously a mix of ASP and HTML) and is now a WordPress site running on nginx. I submitted the sitemap to ...
- 153
2
votes
1
answer
473
views
Robots.txt for a multilanguage site where root is redirected
I have a site which offers two languages, English and Spanish. When the user navigates to the home page, let's say www.example.com the page redirects you to either /es if your browser language is ...
3
votes
1
answer
88
views
Why don't SEO tools such as Screaming frog crawl some of my links?
I have several internal links in my inner pages, for example Interlaken, Switzerland. I used several SEO tools including Screaming Frog to generate sitemap.
All SEO tools index links under the "...
- 133
1
vote
1
answer
417
views
Can search engine fetch/index website that using ipv6 only?
I see some host providers offer cheap server but only support ipv6. I want to rent it, but I am still considering the SEO impact.
Can google and other search engine crawl and index it? If yes, will it ...
- 1,136
3
votes
1
answer
164
views
List of potential origins of host domains through which google bot crawls?
What is the potential list of origins of the host domains through which google bot crawls?
e.g.: Like Google cache uses the following domain to crawl the APIs: http://webcache.googleusercontent.com/
I ...
- 133
0
votes
2
answers
61
views
How to know when a new website comes into existence?
Is there a way to know when a new website comes into existence. What I mean is how crawlers such as search engines know if something exists in first place. So let's say a new website www.example.com ...
3
votes
0
answers
91
views
Do HTML elements with the boolean attribute hidden get picked up by search engines?
I recently learned of the existence of the hidden attribute. It basically hides elements that are not relevant in the current state of the webpage (e.g., a user that is not logged in):
<p hidden>...
- 133