This is something that actually drove me slightly insane not long ago, I couldn't find anyone giving an accurate answer, almost like it was closely guarded secret ;) and hundreds of people looking for a straight answer and a code snippet to do so.
You are correct however that Google will pay attention to the modified date if its the only date/time on the page, therefore you need to only show the modified date when there is one. Giving you the "freshness" capability when revisiting and updating posts/pages.
Since you haven't mentioned what type of site/CMS you are using I'm going to go ahead and assume you are using Wordpress.
The code below you will need to replace your date/time stamp in your current template, this could be different depending on what theme you are using. Generally in single.php , loop-single.php or byline.php
<?php if (get_the_modified_time() != get_the_time()) : ?>
Updated On: <?php the_modified_time('F j, Y'); ?> at <?php the_modified_time('g:i a'); ?>
<?php else: ?>
Posted on <?php the_time('F j, Y'); ?> at <?php the_time('g:i a'); ?>
<?php endif; ?>
Source : https://www.rivmedia.co.uk/how-to-display-the-last-modified-date-of-your-posts-in-wordpress-for-seo-freshness/4902
You can also use search console to "Fetch as Google bot" to speed up indexing of the modified pages.
Happy modifying.