The short list:
- Browser plugins. ActiveX* in general and Flash in particular are notorious for having holes.
- Other errors. I recall bugs in the past where the browser could be tricked into downloading files into a known location, then execute them.
*Google is working on expanding this particular kind of hole to other browsers with Native Client.
Browser bugs and misconfiguration can allow sites that should be in the "Internet" (secure) security zone execute code as if they were trusted. They can then use ActiveX components to install malware.
Once a bug is exploited, the procedure is to inject "shell code" (a chunk of native compiled code), into the process memory to get executed.