Chrome's incognito window uses a fresh cache, so you could say that Chrome has two caches - a "normal" cache and an "incognito" cache. When you close ALL incognito windows and tabs, the incognito cache gets cleared. With default Chrome settings, the normal cache persists across windows and browsing sessions.
It takes extra time for your site to load in incognito because your browser is re-downloading a lot of the page's resources that it would otherwise have already cached, because it's working with a fresh incognito cache.
You can improve load times for repeat visitors by making sure your site's resources are being served with long-life cache headers where appropriate, so the browser won't re-download them for a long time.
loading="lazy" image attribute or
You can improve load times for both by making sure your server hardware is fast (i.e. not on over-provisioned shared hosting) and your server software is lean (e.g. not bogged down by too many plugins).