Each Page that has its own content should get its own canonical URL. It sounds like each of the pages your mention have their own content and should get their own canonical URL:
/ -> https://example.com/
/portfolio -> https://example.com/portfolio
/portfolio/work-1 -> https://example.com/portfolio/work-1
/blog -> https://example.com/blog
Infinite scroll has no direct impact on SEO because Googlebot does not scroll the page. It won't matter if you change the URL on scroll or not for crawlability, because Googlebot does not crawl that way. Googlebot loads the page, but does not interact with it at all. The bot doesn't click, scroll, hover, or otherwise interact with the page. Any page ...