I know the algorithms that is used by search engines can't be exactly the same. But do I need to take bing and other search engines into consideration when building a site, or will my site be optimized for them if I just focus on optimizing my site for Google?
Most often optimization for Google means clear page structure, sitemaps, fast page speed, good incoming links, etc. This of course optimizes for other search engines. Nonetheless you should at least look at the other major search engines, most importantly Bing. A good placement at Google is often also a good placement in all other search engines.