I am more familiar with KVM part working on x86 architecture, so try to explain this in KVM s x86 implementation.
In x86 architecture, KVM leverages CPU s functionality to separate hypervisor and guest mode. In Intel terms, they are VMX root and non-root modes respectively.
VM entry (hypervisor -> VM) is fired by KVM with VMLAUNCH instruction with all guest-needed information filled in CPU s VMCS in kernel mode. Only a system call is invoked from qemu-kvm to kvm kernel module.
A VM exit happens while guest OS is handling something that out of its privilege, such as accessing a physical HW or an interrupt happened. After that, a VM entry is issued and CPU changes to non-root mode again to execute guest code. In summary, VM exit (VM -> hypervisor) is done by HW automatically, and the corresponding exit reason and information would be recored in VMCS. KVM then check VMCS to determine its next step. There is no system call for VM -> hypervisor.
Most device emulations are based in userspace where qemu-kvm can leverage the existing qemu s code. However some device passthrough technologies, such as Intel VT-d, allow guest to access hardware directly through IOMMU or others. Which can bring more powerful performance especially on high speed networking devices.
If you want to dig out the source code, I recommend to focus on CPU virtualization (Intel VT-x) first, which is located in linux/arch/x86/kvm/vmx.c. Intel software developer guide also has comprehensive introduction to VT as well.
