# QEMU Starter Guide Pt. 2
---
## Quick Example
---
## Outline
* Recap: User-mode, helpers, and QOM
* Which machine?
* System-mode high-level control flow
* System-mode threads
* Address spaces
* How does our program work?
* Keyboard Input
* Screen output
---
## Recap: User-mode, helpers, and QOM
----
### User-mode
* Entrypoint: `linux-user/main.c`
* Loads binary
* Outer loop:
* Handle interrupts
* Inner loop
* Read executable bytes
* Lift to tinycode
* Lower tinycode to host instructions
* Execute
----
### Inner loop (guest)
```asm
inb $0x60, %al
```
----
### Inner loop (tinycode)
```asm
---- 0000000000000240 000000000000003c 0000000000000000
call inb,$0x0,$1,tmp0,env,$0x60
deposit_i64 rax,rax,tmp0,$0x0,$0x8
add_i64 rip,rip,$0x2
goto_tb $0x0
exit_tb $0x7fa9ec133e00
set_label $L0
exit_tb $0x7fa9ec133e03
```
----
### Inner loop (host)
```asm
-- guest addr 0x0000000000000240 + tb prologue
movl -0x14(%rbp), %ebx
testl %ebx, %ebx
jl 0x7fa9ec133f0c
movb $1, -0x10(%rbp)
movl $0x60, %esi
movq %rbp, %rdi
callq *0x3b(%rip)
movq (%rbp), %rbx
movb %al, %bl
movq %rbx, (%rbp)
movq 0x100(%rbp), %rbx
addq $2, %rbx
movq %rbx, 0x100(%rbp)
nop
jmp 0x7fa9ec133f00
leaq -0x107(%rip), %rax
jmp 0x7fa9ec000018
leaq -0x110(%rip), %rax
jmp 0x7fa9ec000018
```
----
### Inner loop (helper)
```c
target_ulong helper_inb(CPUX86State *env, uint32_t port)
{
return address_space_ldub(&address_space_io, port,
cpu_get_mem_attrs(env), NULL);
}
```
----
### The Qemu Object Model (QOM)
* OOP system in C
* Single-inheritance of types
* Multiple inheritance of interfaces
* Dynamic type registring (modules)
* Map internal members to publicly exposed properties
----
#### Example - Class def.
```c
typedef struct AnimalClass {
ObjectClass parent_class;
// static data
uint32_t static_field;
// virtual function
void (*make_noise(Animal *));
} AnimalClass;
typedef struct Animal {
int num_legs;
} Animal;
static void make_noise(Animal *a) {
printf("Generic %d legged animal noise!\n", a->num_legs);
}
static void animal_class_init(ObjectClass *oc, void *data) {
AnimalClass *ac = ANIMAL_CLASS(oc);
ac->static_field = 1234;
ac->make_noise = make_noise;
}
static void animal_init(Object *obj) {
Animal *a = ANIMAL(obj);
a->num_legs = 0;
}
```
----
#### Example - Derived
```c
typedef struct CowClass {
ObjectClass parent_class;
} CowClass;
typedef struct Cow {
Animal parent_obj;
int num_spots;
} Cow;
static void cow_class_init(ObjectClass *oc, void *data) {
AnimalClass *ac = ANIMAL_CLASS(oc);
ac->make_noise = moo;
}
static void cow_init(Object *obj) {
Animal *a = ANIMAL(obj);
Cow *c = COW(obj);
c->num_spots = rand();
a->num_legs = 4;
}
static void moo(Animal *a) {
Cow *c = COW(a);
printf("Moo from a %d spotted cow!\n", c->num_spots);
}
```
----
#### Example - Type registring
```c
static const TypeInfo animal_type_info = {
.name = TYPE_ANIMAL,
.parent = TYPE_OBJECT,
.instance_size = sizeof(Animal),
.instance_align = ...,
.instance_init = animal_init,
.instance_post_init = ...,
.instance_finalize = ...,
.abstract = ...,
.class_size = sizeof(AnimalClass),
.class_init = animal_class_init,
.class_base_init = ...,
.class_data = ...,
.interfaces = ...,
};
```
----
#### Example - Type registring
```c
static void register_animal_types(void) {
type_register_static(&animal_type_info);
}
type_init(register_animal_types);
```
----
#### Example - After all that
```c
Cow *cow = COW(object_new(TYPE_COW));
Animal *animal = ANIMAL(cow);
// ...
AnimalClass *ac = ANIMAL_CLASS(animal);
ac->make_noise(animal);
// Moo from a 5 spotted cow!
```
---
## Which machine?
----
## Which machine?
```shell
$ qemu-system-x86_64 -accel tcg \
-fda file=./boot.img \
-vga virtio \
-m 256M \
-cpu qemu64
```
----
### Which machine?
* Intel 440fx chipset (1996)
* Pentium II
----
## Oh
<img src="https://pad.rev.ng/uploads/ffffde37-e23a-4db1-88f1-2aa715b364d6.png" height="500px" />
----
### i440fx
<img src="https://pad.rev.ng/uploads/9f1cdaa1-b00b-4285-ab91-396f769b12b1.png" height="500px" />
* https://wiki.qemu.org/File:QEMU-PC.png
----
### i440fx (simpler)
<img src="https://pad.rev.ng/uploads/bed708de-fc1a-44b6-9333-c690db9fc184.png" height="500px" />
* https://wiki.qemu.org/File:QEMU-PC.png
----
### i440fx (QEMU)
```shell
$ qemu-system-x86_64 -monitor stdio ...
(qemu) info qtree
bus: main-system-bus
type System
dev: ps2-mouse, id ""
gpio-out "" 1
dev: ps2-kbd, id ""
gpio-out "" 1
dev: hpet, id ""
gpio-in "" 2
gpio-out "" 1
gpio-out "sysbus-irq" 32
...
```
----
### i440fx (QEMU)
----
### i8042 Device
```c
static const TypeInfo i8042_info = {
.name = TYPE_I8042,
.parent = TYPE_ISA_DEVICE,
.instance_size = sizeof(ISAKBDState),
.instance_init = i8042_initfn,
.class_init = i8042_class_initfn,
.interfaces = (const InterfaceInfo[]) {
{ TYPE_ACPI_DEV_AML_IF },
{ },
},
};
```
----
### i8042 class constructor
```c
static void i8042_class_initfn(ObjectClass *klass, const void *data)
{
DeviceClass *dc = DEVICE_CLASS(klass);
AcpiDevAmlIfClass *adevc = ACPI_DEV_AML_IF_CLASS(klass);
device_class_set_props(dc, i8042_properties);
device_class_set_legacy_reset(dc, i8042_reset);
dc->realize = i8042_realizefn;
dc->vmsd = &vmstate_kbd_isa;
adevc->build_dev_aml = i8042_build_aml;
set_bit(DEVICE_CATEGORY_INPUT, dc->categories);
}
```
----
### i8042 properties
```c
static const Property i8042_properties[] = {
DEFINE_PROP_BOOL("extended-state", ISAKBDState, kbd.extended_state, true),
DEFINE_PROP_BOOL("kbd-throttle", ISAKBDState, kbd_throttle, false),
DEFINE_PROP_UINT8("kbd-irq", ISAKBDState, kbd_irq, 1),
DEFINE_PROP_UINT8("mouse-irq", ISAKBDState, mouse_irq, 12),
};
```
----
### i8042 instance constructor
```c
static void i8042_initfn(Object *obj)
{
ISAKBDState *isa_s = I8042(obj);
KBDState *s = &isa_s->kbd;
memory_region_init_io(isa_s->io + 0, obj, &i8042_data_ops, s,
"i8042-data", 1);
memory_region_init_io(isa_s->io + 1, obj, &i8042_cmd_ops, s,
"i8042-cmd", 1);
object_initialize_child(obj, "ps2kbd", &s->ps2kbd, TYPE_PS2_KBD_DEVICE);
object_initialize_child(obj, "ps2mouse", &s->ps2mouse,
TYPE_PS2_MOUSE_DEVICE);
qdev_init_gpio_out(DEVICE(obj), s->irqs, 2);
qdev_init_gpio_in_named(DEVICE(obj), i8042_set_kbd_irq,
"ps2-kbd-input-irq", 1);
qdev_init_gpio_in_named(DEVICE(obj), i8042_set_mouse_irq,
"ps2-mouse-input-irq", 1);
}
```
----
### i8042 instance realization
```c
static void i8042_realizefn(DeviceState *dev, Error **errp)
{
...
isa_connect_gpio_out(isadev, I8042_KBD_IRQ, isa_s->kbd_irq);
isa_connect_gpio_out(isadev, I8042_MOUSE_IRQ, isa_s->mouse_irq);
isa_register_ioport(isadev, isa_s->io + 0, 0x60);
isa_register_ioport(isadev, isa_s->io + 1, 0x64);
...
}
```
---
## System-mode high-level control flow
----
### System-mode entry point
```c
// system/main.c
int main(int argc, char **argv)
{
qemu_init(argc, argv);
...
qemu_default_main(NULL);
}
```
----
### CLI Parsing and initialization
```c
// system/vl.c
void qemu_init(int argc, char **argv)
{
// CLI parsing
...
// ->
qemu_create_machine(machine_opts_dict);
// e.q. accel/tcg
configure_accelerators(argv[0]);
qemu_init_displays();
}
```
----
### Machine init
```c
// hw/i386/pc_piix.c
static void pc_i440fx_init(MachineState *machine)
{
pc_init1(machine, TYPE_I440FX_PCI_DEVICE);
}
```
----
### Machine init
```c
// hw/i386/pc_piix.c
static void pc_init1(MachineState *machine, const char *pci_type)
{
...
// Create initialize vCPUs
x86_cpus_init(x86ms, pcmc->default_cpu_version);
// PCI
pci_memory = g_new(MemoryRegion, 1);
memory_region_init(pci_memory, NULL, "pci", UINT64_MAX);
pcms->pcibus = PCI_BUS(qdev_get_child_bus(DEVICE(phb), "pci.0"));
// Initializes RAM memory regions
// Loads BIOS, sets fw_cfg
pc_memory_init(pcms, system_memory, pci_memory, hole64_size);
// ISA bus
isa_bus = ISA_BUS(qdev_get_child_bus(DEVICE(pci_dev), "isa.0"));
if (x86ms->pic == ON_OFF_AUTO_ON || x86ms->pic == ON_OFF_AUTO_AUTO) {
pc_i8259_create(isa_bus, gsi_state->i8259_irq);
}
// PIC and VGA
ioapic_init_gsi(gsi_state, phb);
pc_vga_init(isa_bus, pcms->pcibus);
// Initializes floppy controller
// loads our program image
pc_basic_device_init(pcms, isa_bus, x86ms->gsi, x86ms->rtc,
!MACHINE_CLASS(pcmc)->no_floppy, 0x4);
}
```
----
### vCPU realization
```c
// target/i386/cpu.c
static void x86_cpu_realizefn(DeviceState *dev, Error **errp)
{
...
qemu_init_vcpu(cs);
cpu_reset(cs);
}
```
----
### vCPU thread creation
```c
//system/cpus.c
void qemu_init_vcpu(CPUState *cpu)
{
...
cpus_accel->create_vcpu_thread(cpu);
}
```
----
### vCPU thread loop
```c
// accel/tcg/tcg-accel-ops-mttcg.c
static void *mttcg_cpu_thread_fn(void *arg)
{
...
do {
qemu_process_cpu_events(cpu);
bql_unlock();
r = tcg_cpu_exec(cpu);
bql_lock();
} while (!cpu->unplug || cpu_can_run(cpu));
}
```
----
### What happens now?
* BIOS is executed
* ...
* Load our program at 0x7c00
* Jumps to 0x7c00
----
## System-mode threads
----
### System-mode threads
* 1 main event loop thread
* N vCPU threads
* ~1 large mutex (BQL)
----
### System-mode entry point
```c
// system/main.c
int main(int argc, char **argv)
{
qemu_init(argc, argv);
...
qemu_default_main(NULL);
}
```
----
### Main thread event loop
```c
// system/runstate.c
int qemu_main_loop(void)
{
int status = EXIT_SUCCESS;
while (!main_loop_should_exit(&status)) {
main_loop_wait(false);
}
return status;
}
```
----
### Main thread event loop
```c
// util/main-loop.c
void main_loop_wait(int nonblocking)
{
// Waiting for events, e.g. keyboard
ret = os_host_main_loop_wait(timeout_ns);
// Notify timer updates, e.g. VGA
qemu_clock_run_all_timers();
}
```
----
### Main thread event loop
```c
// util/main-loop.c
static int os_host_main_loop_wait(int64_t timeout)
{
bql_unlock();
ret = qemu_poll_ns((GPollFD *)gpollfds->data, gpollfds->len, timeout);
bql_lock();
}
```
----
### vCPU thread loop
```c
// accel/tcg/tcg-accel-ops-mttcg.c
static void *mttcg_cpu_thread_fn(void *arg)
{
...
do {
qemu_process_cpu_events(cpu);
bql_unlock();
r = tcg_cpu_exec(cpu);
bql_lock();
} while (!cpu->unplug || cpu_can_run(cpu));
}
```
---
## Address spaces
----
### Address spaces
* If you don't understand the control flow in QEMU, it's usually either:
* helper functions, or;
* address space callbacks
----
### Memory Regions
```c
struct MemoryRegion {
// MMIO read/write callbacks
const MemoryRegionOps *ops;
// parent region
MemoryRegion *container;
// Occupied address range
hwaddr addr;
Int128 size;
// Children
QTAILQ_HEAD(, MemoryRegion) subregions;
};
```
----
### Memory Region Types
```c
// Container, no backing memory
memory_region_init()
// Backed by host memory
memory_region_init_ram()
// I/O registers calls ops->read()/write
memory_region_init_io()
// View of another region
memory_region_init_alias()
```
----
### Memory Regions
----
### `FlatView`s
```c
// include/system/memory.h:1194
struct FlatView {
// Array of ranges
FlatRange *ranges;
unsigned nr;
// Radix tree for quick lookups (page tables)
struct AddressSpaceDispatch *dispatch;
// Tree
MemoryRegion *root;
};
// system/memory.c:222
struct FlatRange {
// Backing region
MemoryRegion *mr;
hwaddr offset_in_region;
// Corresponding range in flattened space
AddrRange addr;
};
```
----
### `FlatView`s
----
### `FlatView`s
----
### `FlatView`s
* Memory regions can have different priorities
* Whenever memory regions change
* Rebuild flat views
* Rebuild radix tree
* Notify listeners
* e.g. flush TLBs
----
### `AddressSpace`s
```c
struct AddressSpace {
MemoryRegion *root;
struct FlatView *current_map;
QTAILQ_HEAD(, MemoryListener) listeners;
};
```
----
### Global `AddressSpace`s
```c
static void memory_map_init(void)
{
memory_region_init(system_memory, NULL, "system", UINT64_MAX);
address_space_init(&address_space_memory, system_memory, "memory");
memory_region_init_io(system_io, NULL, &unassigned_io_ops, NULL, "io", 65536);
address_space_init(&address_space_io, system_io, "I/O");
}
```
---
## How does our program work?
----
### How does our program work?
* Bootloader stage 1
* Bootsector headers
* Loads stage 2
* Bootloader stage 2
* Loads program
* Gets VESA framebuffer from BIOS
* Set up page-table
* Switch to 64-bit long mode
* Jump to program
----
### How does our program work?
* Actual program
* Set up Programmable Interrupt Controller (PIC)
* Register
* Interrupt Service Routines (ISRs)
* Interrupt Request Handlers (IRQs)
* When a ps/2 keyboard interrupt occurs, draw a character to the VESA framebuffer
----
### What happens when I press a key?
* Main thread gets key event from SDL
* Main thread notifies listeners
* ps/2 controller responds, stores key
* ps/2 controller raises PIC IRQ 1
* PIC notifies CPU
* CPU redirects execution to guest
----
## Keyboard input
----
### Interfacing with the ps/2 controller
```asm
; reg_cmd = 0x64
; reg_data = 0x60
; configure
outb 0x64, 0x20 ; get-config
inb al, 0x60
...
; when keyboard interrupt occurs
inb al, 0x60
```
----
### i8042 (again)
```c
static void i8042_initfn(Object *obj)
{
memory_region_init_io(isa_s->io + 0, obj, &i8042_data_ops, s,
"i8042-data", 1);
memory_region_init_io(isa_s->io + 1, obj, &i8042_cmd_ops, s,
"i8042-cmd", 1);
}
```
----
### i8042 (again)
```c
static void i8042_realizefn(DeviceState *dev, Error **errp)
{
isa_register_ioport(isadev, isa_s->io + 0, 0x60);
isa_register_ioport(isadev, isa_s->io + 1, 0x64);
}
```
----
### i8042 (again)
```c
void isa_register_ioport(ISADevice *dev, MemoryRegion *io, uint16_t start)
{
memory_region_add_subregion(isa_address_space_io(dev), start, io);
}
```
----
### Reading keyboard input (source)
```zig
pub fn handler(registers: *idt.Registers) void {
console.log_fmt("scancode: {x}", .{x86.inb(reg_data)});
}
```
----
### Reading keyboard input (guest)
```asm
inb $0x60, %al
```
----
### Reading keyboard input (tinycode)
```asm
---- 0000000000000240 000000000000003c 0000000000000000
call inb,$0x0,$1,tmp0,env,$0x60
deposit_i64 rax,rax,tmp0,$0x0,$0x8
add_i64 rip,rip,$0x2
goto_tb $0x0
exit_tb $0x7fa9ec133e00
set_label $L0
exit_tb $0x7fa9ec133e03
```
----
### Reading keyboard input (host)
```asm
-- guest addr 0x0000000000000240 + tb prologue
movl -0x14(%rbp), %ebx
testl %ebx, %ebx
jl 0x7fa9ec133f0c
movb $1, -0x10(%rbp)
movl $0x60, %esi
movq %rbp, %rdi
callq *0x3b(%rip)
movq (%rbp), %rbx
movb %al, %bl
movq %rbx, (%rbp)
movq 0x100(%rbp), %rbx
addq $2, %rbx
movq %rbx, 0x100(%rbp)
nop
jmp 0x7fa9ec133f00
leaq -0x107(%rip), %rax
jmp 0x7fa9ec000018
leaq -0x110(%rip), %rax
jmp 0x7fa9ec000018
```
----
### Reading keyboard input (QEMU)
```c
/* target/i386/tcg/system/misc_helper.c:37 */
target_ulong helper_inb(CPUX86State *env, uint32_t port)
{
return address_space_ldub(&address_space_io, port,
cpu_get_mem_attrs(env), NULL);
}
```
----
### Reading keyboard input (QEMU)
```c
address_space_ldub()
address_space_ldm_internal()
address_space_translate()
memory_region_dispatch_read()
memory_region_read_accessor()
mr->ops->read()
kbd_read_data()
ps2_read_data()
```
---
### Screen output
----
```zig
pub fn handler(registers: *idt.Registers) void {
console.log_fmt("scancode: {x}", .{x86.inb(reg_data)});
}
```
----
### Screen output
```zig
fn pixel(x: usize, y: usize, color: u32) void {
framebuffer[width * y + x] = color;
}
```
----
### VGA device init
```c
// hw/display/vga.c:2194
bool vga_common_init(VGACommonState *s, Object *obj, Error **errp)
{
memory_region_init_ram_flags_nomigrate(&s->vram, obj, "vga.vram",
s->vram_size, 0, &local_err);
vmstate_register_ram(&s->vram, s->global_vmstate ? NULL : DEVICE(obj));
s->vram_ptr = memory_region_get_ram_ptr(&s->vram);
/*
* Set default fb endian based on target, could probably be turned
* into a device attribute set by the machine/platform to remove
* all target endian dependencies from this file.
*/
s->default_endian_fb = target_big_endian();
s->big_endian_fb = s->default_endian_fb;
vga_dirty_log_start(s);
return true;
}
```
----
### BIOS
* Sets up PCI BAR at a common address, usually 0x7d000000
* 0x7d000000 gets read by our program and as `framebuffer`
----
### Screen output (guest)
```asm
movl $0xff00ff, (%r13, %r15, 4)
```
----
### Screen output (tinycode)
```asm
---- 000000000000054a 000000000000000b 0000000000000000
shl_i64 tmp2,r15,$0x2
add_i64 tmp2,tmp2,r13
qemu_st_i64 $0xff00ff,tmp2,noat+un+leul,4
```
----
### Screen output (host)
```asm
-- guest addr 0x000000000000054a
; TLB check
movq 0x78(%rbp), %r14
leaq (, %r14, 4), %r15
addq %r15, %rbx
movl $0xff00ff, %r15d
movq %rbx, %rdi
shrq $7, %rdi
andq -0x80(%rbp), %rdi
addq -0x78(%rbp), %rdi
leaq 3(%rbx), %rsi
andq $0xfffffffffffff000, %rsi
cmpq 8(%rdi), %rsi
; slow path
jne 0x7fa9ec1e1934
; TLB hit
movq 0x18(%rdi), %rdi
movl %r15d, (%rbx, %rdi)
```
----
### (Finally) Screen output
* VGA controller only needs to update screen for dirty pages
* Main thread triggers screen refresh (timer)
* On screen refresh all dirty pages are cleaned
* First write to pages will be slow
---
## Summary
* 1 Main event thread
* N vCPU threads
* machine modeled in QOM types
* memory regions link together devices
---
## Questions?