but now I have the awkward situation that I can access matomo.example.com at example.com/example.com/matomo, which will confuse it since the hostname does not match.
In .htaccess you can check the hostname and redirect to the subdomain if the site has been erroneously accessed via the subdirectory on the main domain.
For example, in the .htaccess file located in the document root of the subdomain (which I assume is at example.com/example.com/matomo/.htaccess) you can do something like the following near the top of the file using mod_rewrite:
RewriteEngine On
RewriteCond %{HTTP_HOST} !=matomo.example.com
RewriteRule ^ https://matomo.example.com%{REQUEST_URI} [R=301,L]
This has the added benefit that it canonicalises any malformed request that contains a mixed-case hostname or a FQDN with a trailing dot.
OR, you can do something similar in the parent .htaccess file that affects all subdomains (accessed via the subdirectory) - but that would be dependent on whether the subdomain already has its own .htaccess file that uses mod_rewrite (the subdomain's mod_rewrite directives will override the parent directives by default).
Is there something that I can do with .htaccess and mod_rewrite such that the .htaccess only applies to the www part and also that I have independent subdirectories?
.htaccess files work along the filesystem path. The .htaccess file in the parent directory will naturally control the entire directory tree from the parent down through all subdirectories, so it is "tricky" to have true independence (without access to the server config).
However, different Apache modules behave differently in terms of how the directives are inherited. With mod_rewrite, providing the inheritance model has not been changed in the server-config, then simply enabling the rewrite engine (ie. RewriteEngine On) in the subdirectory's .htaccess file is sufficient to completely override any mod_rewrite directives in any parent .htaccess file. But directives from other modules cannot be overridden so easily (provision would need to be made in the parent .htaccess file).
Workaround
Instead of serving your main-site from the document root of the main domain, you could move your main-site to a subdirectory (effectively mimicking your original setup) and rewrite all requests for the domain apex or www subdomain to this subdirectory.
The document root of your main domain would then have minimal files, perhaps just a simple .htaccess file that rewrites all requests. For example, assuming all your main-site files are in a /mainsite subdirectory then you could do something like the following:
RewriteEngine On
RewriteCond %{HTTP_HOST} ^(www\.)?example\.com [NC]
RewriteRule !^mainsite/ /mainsite%{REQUEST_URI} [L]
UPDATE: Unless you have additional .htaccess files (that contain mod_rewrite directives) in the subsite's subdirectories then the above is sufficient to prevent the subsites from being accessed by their respective subdirectories off the main domain, since (as you mentioned in comments) the request is internally rewritten to the /mainsite subdirectory - which will probably result in a 404.
However, if the subsite does use its own .htaccess file with its own mod_rewrite directives then - by default - these will completely override the above directives in the document root, so the erroneous request for the subdirectory will not be rewritten to the mainsite. In this case, either implement the redirect as mentioned in the first part of this answer, OR if you are using Apache 2.4+ then you can enable mod_rewrite inheritance in the parent .htaccess file so that the parent mod_rewrite directives are processed first. You can do this by adding the RewriteOptions InheritDownBefore directive to the .htaccess file in the document root. This then becomes (with additional comments):
RewriteEngine On
# Apache 2.4+ Execute these directives BEFORE child configs
RewriteOptions InheritDownBefore
# Rewrite all requests for the main domain to a subdirectory
RewriteCond %{HTTP_HOST} ^(www\.)?example\.com [NC]
RewriteRule !^mainsite/ /mainsite%{REQUEST_URI} [L]
example.com/example.com/matomo" - is the repetition ofexample.comintentional?