I resolved the issue. The solution is really simple but worked for me.
What I did is, as I was using Amazon and flipkart static banners, I downloaded those banners and uploaded them to my website and linked them with my https affiliate link.
Because the issue was with banners only which were served over http, as I checked the code provided by them.
I understand from your question that you don't think it's pulling from cache. I'm not sure why you believe that or if I'm not understanding your question
According to https://developer.mozilla.org/en-US/docs/Web/HTTP/Status/304, the HTTP 304 status code indicates
an implicit redirection to a cached resource.
Many developer tools' network ...