All Questions
Tagged with google web-crawlers
110 questions
0
votes
0
answers
12
views
Why aren't search engines showing my site anymore? [duplicate]
I have created a site as a home project but for some reason, it stopped showing my site in Google. I connected my site to Google Search Console and most of the pages are indexed. At least I can see it ...
- 1
1
vote
1
answer
25
views
How can a react/vue apps like these rank just by using client side rendering?
By client-side rendering I mean, you can't see any content (h1,title,meta tags etc) in the source code, which is the only place where google bot can learn what a page is about.
here is this site: ...
- 11
3
votes
2
answers
5k
views
What does User-agent: Mediapartners-Google mean?
I was going through the google support website and there they said:
"To update your robots.txt file to grant our crawler access to your pages, remove the following two lines of text from your ...
- 102
4
votes
2
answers
1k
views
How are 'scraped content' websites like diningandcooking.com able to rank so well despite having no original content?
I've stumbled upon a website called https://diningandcooking.com after finding out it was linking to one of my websites. When i went to the page that had the link on it i found out that its content ...
- 387
2
votes
1
answer
996
views
Can we request google to crawl our website every single day to update its indexes and tags on our domain?
I m thinking of making a blog website and noticed that google search results appear as per Google's indexing upon the crawled website. And this web crawling by google happens like once in 2 weeks to a ...
- 21
8
votes
1
answer
791
views
Are 'robots.txt' rules "starts with" or "contains" rules (clarification on confusing Google documentation)
Searching for robots.txt pattern matching rules (inc. the non official Google rules), I found this Google dev page where it says
the /fish rule matches any path that starts with /fish (...) it doesn'...
- 183
11
votes
4
answers
6k
views
Prevent Google from thinking my own bot is faking traffic on my website
I have a large website which I want to be visited by my own bot early on the morning, every day (for caching purpose). I want my bot visits 10,000 pages per day.
My natural traffic is about 5000 pages ...
- 111
1
vote
1
answer
397
views
Should I add links to the Sitemap that are disallowed in robots.txt?
My robots.txt contains the line Disallow: /search.php.
Shall I nevertheless include http://example.com/search.php into my sitemap?
- 113
0
votes
1
answer
322
views
HTML sitemap with noindex, follow: Matt Cutts vs John Mueller
Ref: Do noindex,follow pages pass pagerank / link juice?
I have a requirement wherein I want to create an HTML sitemap to help users navigate to deep pages. This sitemap page will be linked from ...
- 2,343
0
votes
0
answers
41
views
How to block google from indexing mail record. duplication issue mail.example.com which is duplication of example.com
google is causing duplication by indexing mail.example.com which is identical to home page of www.example.com/
I added property in search console and removed this subdomain but keeps on indexing ...
1
vote
1
answer
448
views
I have 2.1 million pages, how to get crawled by googlebot more frequently? [duplicate]
I have a project which aggregates public information on companies, startups, founders etc. for a total of >2 million tracked entities worldwide.
Yes, every single page is high quality, manually ...
- 111
0
votes
0
answers
187
views
Using prerender.io for SEO on my AngularJS app - Dynamic and static web pages
I have a website that was built with AngularJS. I want the website to be crawled by search engines (with the dynamic content).
I've tried to use Google services to do the crawl (from a sitemap.xml ...
1
vote
0
answers
10
views
Should we apply "noreferrer" on "Read More" or "View All"?
I experienced years back a search engine crawler issue that Google started showing "Read More" as title of the page in it's search results. Because that page was linked through "read more" link on new ...
1
vote
1
answer
71
views
Robot.Txt File interpretation problem?
I have a Joomla/Shopware For an eCommerce website, but I see some problems and I think google didnt crawl pages:
{block name="frontend_robots_txt_user_agent"} User-agent: * {/block}
{block name="...
- 141
2
votes
1
answer
3k
views
Do I need to disallow mediapartners-google in robots.txt for SEO?
I found that most of the websites disallow mediapartners-google in robots.txt.
Why do I need to do block mediapartners-google? Can blocking it help withp better SEO or does it not matter for SEO?
- 155
0
votes
0
answers
98
views
Google is marking as duplicate my unique URLs powered by Angular Universal
I have a problem in the search console, and google is marking my unique urls as duplicate
The application is build with Angular Universal and is rendered in the server
Note: This site is only ...
1
vote
0
answers
81
views
When using an image server on a separate domain, should we link directly to the image URL and restrict crawling, or redirect through our domain?
I have a large number of images stored on a separate server, these URLs currently go through a 302 redirect. I think these additional pages are a total waste of time for Google to crawl.
At the ...
- 49
1
vote
2
answers
344
views
Deleted tags & categories from my blog still come up in Google search results
I am a newbie in blogging. I've deleted some of the unwanted tags and categories from my wordpress blog but they kinda still come put in the search results when I search my website on Google.
How do ...
- 133
0
votes
1
answer
23
views
How deep do search engine crawlers go?
I have 2 sites with very similar HTML structure and I would like to know how google or any other crawler rank these 2 sites.
Will they penalize me due the sites sharing a structure?
The content ...
- 3
2
votes
0
answers
15
views
Prevent bad rankings on a Q and A site for questions without an answer by using noindex until answered [duplicate]
I have a website on which users can ask a question & also they can answer it.
So, when a user asks a question, it creates a new web-page on the website & below that question lies it's answers ...
user89321
3
votes
0
answers
293
views
Does Googlebot cache JavaScript files between two pages if they expire in one year?
My question is about javascript websites.
Suppose GoogleBot has crawled example.com/index.html?id=1
Does it request all the javascript files again when crawling example.com/index.html?id=2?
index....
- 243
2
votes
2
answers
4k
views
Would multiple images with the same alt text as the page name be keyword stuffing?
In a page I have three or four images that have same alt tag as of my page name. Could Google consider that to be keyword and penalize my site?
Is it good practice to include page name in alt tag?
...
- 31
1
vote
1
answer
396
views
Website Home Page URL Not Show On Google Search [duplicate]
If i search like site:www.paka.tv my main homepage URL not showing, but categories of my site pages showing on results. How to show my website URL (www.pakka.tv) on google search.
- 181
3
votes
1
answer
3k
views
How to find out the referrer of Googlebot's crawling URL?
Googlebot crawls 100s of 404 URLs from my website. I want to know from where it gets those links?
Is there anything like HTTP Referrer?
In detail: Googlebot crawls
example.com/no_such_files.php
...
- 1,327
1
vote
0
answers
29
views
confused with google url indexing [duplicate]
When enter my site in google like this site:mywebsite.com I have seen this below image.
I have 169 total urls in my sitemap.xml file. It says 102 results found. But, when I go to next page (second ...
- 171
3
votes
1
answer
843
views
How to prevent Googlebot from doing API requests?
I have a currency converter site around 32k pages. Every pair for each page.
And every page has 2 API requests.
I started to see huge number of API requests. From users I have only few page views ...
- 31
1
vote
0
answers
9
views
Wordpress Website not indexing in google, no error on google webmaster [duplicate]
I just recognized my SEO Score is 0 and that my wordpress website is not indexed on google search engine, no automatic filling as well. I am not an expert, I never used a sitemap, nor htaccess but it ...
- 11
1
vote
2
answers
1k
views
How can I stop Google from indexing "pretty links" external redirects from my WordPress site?
When I search site:[example].com in Google for my blog, the majority of the pages that are being indexed are "pretty links" to external sites.
I created these links using the Pretty Links plugin so ...
- 11
2
votes
1
answer
287
views
Should I prevent search engines from indexing empty user profile pages? If so, how much content is enough for indexing?
I'm developing a social website for book readers, with public user profile pages. For each user, there are several pages available:
The main user page (about me, last activity, ...)
Several book ...
- 281
1
vote
0
answers
27
views
Is it bad for SEO to have a URL with nothing on it? [duplicate]
I have a WordPress site and for a very complicated reason I have to set some post type's template to empty file. Nothing.
Which means I have a URL on my site (let's say: example.com/postype/the-best-...
- 21
3
votes
1
answer
543
views
Rule out third party scraping, but allow Google crawling
How to make scraping of own content through wget, httrack etc. impossible, but allow crawling through googlebot?
This should be done without showing to googlebot other content, as to other user ...
- 10.2k
1
vote
1
answer
34
views
How do you properly SEO-tag an app that is only a catalogue of images?
I am making a little helper app that mostly links a lot of graphs and images. There is also some text explaining a couple of things, but more importantly that text will probably never (or very rarely) ...
- 319
3
votes
2
answers
162
views
Google claims not to find my robots.txt
Like I said in the title, the Google Webmaster Console claims not to find the robots.txt file of my homepage and therefore it won't crawl it.
But there is a robots.txt file (https://soundhaus-band....
- 31
1
vote
1
answer
230
views
Noncrawlable Javascript Pagination Issue
We have just completed a revamp of a website with a new design and I had noticed something when running the sitemap generator for the new site. The new site uses javascript pagination which fits the ...
- 31
4
votes
1
answer
805
views
GoogleBot crawling a page up to 3,000 times per day
For a website, GoogleBot daily crawls one particular URL more than one thousand times (mainly between 1,500 and 3,000, sometimes much more).
The Googlebot IP is a real Google one (for example : 66....
1
vote
0
answers
68
views
Dealing with Google spidering JavaScript incorrectly
Google spiders JavaScript now to look for possible URLs. This is mostly a good thing, but occasionally it gets it wrong. Is there any way to hide possible URLs in JavaScript from googlebot?
For ...
- 111
1
vote
1
answer
149
views
Making google forgive us right away when accidentally publishing incorrect links (noarchive doesn't work)
So this is what happened.
One day, I changed my code on my site to try to make things more compliant with adsense. Several hours later, I check the crawl errors report GSC only to see a string of ...
- 13.6k
3
votes
1
answer
699
views
How to disallow Google crawling out of my website through images?
I'm taking care of a website which has multiple images coming from exterior links.
The thing is, those links are outgoing ones, and I want to avoid that as much as possible.
I tough of one solution ...
- 133
3
votes
1
answer
6k
views
Can I safely block Google AdsBot from spidering specific links?
My question is similar to Google Adsbot suddenly crawling odd pages, but my use case is different.
I'm running a price comparison website. I have ads directing users to my home page, and other ads ...
- 281
1
vote
1
answer
277
views
Calculating requests per second within bing
In Google Search Console, I am able to set the rate at which google crawls my site and it tells me the details such as the requests per second and seconds between requests. In Bing Webmaster tools, I ...
- 13.6k
2
votes
2
answers
73
views
How to tell search engines to lower the robots.txt scanning rate on obsolete subdomains?
After putting my site into one domain instead of multiple subdomains, I checked my logs and noticed that search engines (particularly google) are scanning robots.txt several times a day for each ...
- 13.6k
1
vote
1
answer
23
views
What is a good approximation for a page's (Google) search rank to come back to it's original rank, after it has fallen due to technical errors?
Say we had a web page, which initially had a google rank on the first page.
Say these errors occurred after some time:
load time
Meta title changes
Some data not being visible to the crawler
etc.
...
- 689
2
votes
1
answer
383
views
How to exclude Bots from Google Analytics (sudden rise of them in the last weeks)
Since a few weeks, one of my Google Analytics Accounts shows a continued rise of Visitors and Pageimpressions what seams to be bots. They have a Average Session Duration of «<00:00:01» and by now ...
- 141
3
votes
1
answer
126
views
Does site structure/URL structure impact SEO? [duplicate]
We would like to change the following URLs as follows:
(old versions to the left, being changed to new versions on the right)
www.ex.com/xxx/full-seo-term -> www.ex.com/half-seo-term1/half-seo-term2 -...
- 689
2
votes
1
answer
475
views
Is it possible to make a crawlable AJAX web application WITHOUT using HTML snapshots?
Is it possible to make single page application (SPA) crawlable without HTML snapshots with help of HTML5 History API pushState?
I have found contradictory meanings. Can you support or refute one of ...
- 119
2
votes
0
answers
765
views
Fetch as Google not rendering single-page application (AngularJS) - how to show console logs output
I have an SPA application, and it is not being rendered correctly by Google. I tried going to Google Web Master toolkit, and use the Fetch as google feature. This is showing just the HTML output ...
- 153
0
votes
2
answers
699
views
Stop Google from crawling proxied pages but still allow the proxy itself to be found via search engines
I have a proxy at http://rahul2001.com/proxy/
I site can be 'proxied' by like this:
http://rahul2001.com/proxy/proxy.php/http://site-to-be-proxied.example.com/
The problem is that Google seems to be ...
- 111
3
votes
3
answers
80
views
How to properly tell Google they have no right to access the download. What status do I use?
I'm looking for the SEO way to tell Google not to download pictures.
I have a script that generates one of two things based on who accesses the site:
If an unauthorized robot accesses the site, they ...
- 13.6k
2
votes
1
answer
417
views
Website being bombarded by non-existent URLs
I noticed a newly registered domain is being hit by non-existent URLs like
/pithy/597/47363-1117.doc
/pithy/597/47363-1117.doc
/2015/150728.html
etc...
It's being hit for thousands per day which is ...
- 309
2
votes
2
answers
146
views
Disallow Crawling of All Search Pages
Using robots.txt I am attempting to stop all crawling of search URLs
Disallow: /rest_of_url/search&tour*
Above is what I am using. Our URL looks like the following for all search results. ...
- 253