I want to know about video SEO. How does a search engine determine the relevance of a video to a search query? Do they look at individual frames of the video, or does it determine the popularity of the website or the domain where the video is hosted for ranking?
Each search engine is different, but all of the factors you mention are important. More important is how you organize your content with the video. Everything matters from your sitemap, markup, etc. There are a lot details around this very subject.
Google clearly defines standards and recommendations here:
How does a search engine determine the relevance of a video to a search query?
They simply checkout the title and description of video and keywords, and other thing is secret from us.
Do they look at individual frames of the video?
Not at all, Google not looking into videos frame as much deeper, but they can find some other content easily like pornography. Make sure your video has clean audio.
does it determine the popularity of the website?
Well popularity sometimes help to rank better, because it will help you to user spend more time in your site, also people like to share video on social media, so indirectly Google+ share will help you to rank better ( Because If any of user log in with him/her Gmail account then he/she can see result on search page that is shared by him/her friends)
domain where the video is hosted for ranking?
Yes, it's very important. Suppose you hosted some video on YouTube, then your video will crawl many times, Just checkout YouTube PageRank (Reputation), It will help you to rank better easily. Now same video if you hosted on other site, then may be your video will not crawl many time by spider. also reputation of that page is too low. So obviously you rank better with YouTube video.
Google cannot rank a video but can take clues from code, content, and other activities such as that derived through normal means for a page. This is how Google handles images. I cover this in a short answer here: Does Google OCR and index text in the images?. In short, it is a combination of the mark-up surrounding the image/video, any content below the image/video, the
description meta-tag, and
h1 tag that give clues to a search engine.
Also consider the page performance as an issue. From a search standpoint, SERPs impressions verses click-throughs, time spent on page, bounce-rate, etc. are all factors that can give clues to the popularity of the page and can signal the value of any embedded video.
While it is technically possible to extract text from an image thus also a video, as well as transcribe some audio from a video, Google does not do this. It is not only unnecessary, but costly to do.
Please remember that it is often the simplest things that reduce the signal-to-noise ratio in search. Do not over-complicate your thinking. Keep it simple and you will always come out ahead.