Topic 01: Bare-Metal Boot, GDT & TSS Double Fault Stack Isolation

Focus: Boot sequence, Global Descriptor Table initialization, Task State Segment structure, Interrupt Stack Table (IST 0), and hardware-enforced double fault stack switching to prevent triple fault resets.

1. High-Level Summary Table

2. Spoken Interview Answer (Word-for-Word Script)

"When bringing up an x86_64 kernel on bare metal, managing CPU segment descriptors and exception stacks is the first critical defense against unrecoverable hardware resets.

Although 64-bit long mode disables most legacy memory segmentation, the CPU still mandates a valid Global Descriptor Table (GDT) to define code segment privileges and to hold the descriptor for the Task State Segment (TSS). The primary reason we need a TSS in 64-bit mode is not hardware task switching—which was deprecated—but the Interrupt Stack Table (IST).

In standard x86 execution, when an exception occurs, the CPU pushes the return instruction pointer, flags, and stack pointer directly onto the current stack. However, if the kernel encounters a stack overflow, the stack pointer points to an unmapped guard page. Attempting to push the Page Fault exception frame onto this invalid stack immediately triggers a second fault—escalating to a Double Fault (Vector 8). If the double fault handler also uses the current stack, pushing the double fault frame fails as well, causing an instant hardware Triple Fault, which forces the physical motherboard to reboot.

In RustOS, I solved this by configuring entry 0 of the TSS Interrupt Stack Table to point to an isolated, statically allocated 20 KiB emergency stack. In the GDT, I registered both the kernel code segment and the TSS descriptor, reloading CS and the TR register. When a stack overflow occurs, the CPU detects vector 8, automatically switches RSP to the emergency stack pointer defined in IST 0, and safely executes our double fault handler, printing a full diagnostic stack frame instead of rebooting into an infinite reset loop."

3. Deep-Dive Mechanics & Architecture Walkthrough

3.1 Boot Sequence and The entry_point! Macro

The kernel receives execution control from the bootloader crate. In src/main.rs, the entrypoint is established as:

// src/main.rs
entry_point!(kernel_main);

fn kernel_main(boot_info: &'static BootInfo) -> ! {
    println!("Welcome to RustOS, {}!!", "from The Rusty Crew");
    rustos::init();
    // ...
}

The entry_point! macro generates a real C-compatible entry symbol (_start) that extracts the bootloader's BootInfo struct, verifies pointer alignment and stack sanity, and passes a safe &'static BootInfo reference into our Rust function. This eliminates undefined behavior before any kernel code executes.

3.2 GDT and TSS Memory Layout

The Global Descriptor Table is defined lazily using lazy_static! in src/gdt.rs:

// src/gdt.rs
lazy_static! {
    static ref TSS: TaskStateSegment = {
        let mut tss = TaskStateSegment::new();
        tss.interrupt_stack_table[DOUBLE_FAULT_IST_INDEX as usize] = {
            const STACK_SIZE: usize = 4096 * 5; // 20 KB
            static mut STACK: [u8; STACK_SIZE] = [0; STACK_SIZE];
            let stack_start = VirtAddr::from_ptr(&raw const STACK);
            let stack_end = stack_start + STACK_SIZE;
            stack_end // Stacks grow downwards on x86
        };
        tss
    };
}

The TSS contains an array of 7 Interrupt Stack Table (IST) pointers. RustOS configures DOUBLE_FAULT_IST_INDEX = 0 to point to the top (highest memory address) of a 20 KiB static buffer. Because x86 stacks grow downward towards lower memory, stack_end is the correct initial stack pointer.

3.3 Segment Selector Loading

To make the CPU use our GDT and TSS, the descriptors must be written and their selectors loaded into the CPU registers:

// src/gdt.rs
pub fn init() {
    use x86_64::instructions::tables::load_tss;
    use x86_64::instructions::segmentation::{CS, Segment};

    GDT.0.load(); // LGDT instruction
    unsafe {
        CS::set_reg(GDT.1.code_selector); // Reload Code Segment
        load_tss(GDT.1.tss_selector);     // LTR instruction
    }
}

When load_tss(tss_selector) executes, the CPU loads the Task Register (TR) with the selector pointing to the TSS entry in our GDT. The CPU now knows where to find the IST whenever an interrupt gate indicates an IST index.

4. Real Edge Cases, Pitfalls & Debugging

4.1 The Triple Fault Reset Loop

During initial development, a recursive test function was written to test stack boundaries. Without the IST configured, running the test caused QEMU to immediately reboot in a tight loop. Inspecting the QEMU crash log with -d int,cpu_reset revealed:

check_exception old: 0xe new 0x8
check_exception old: 0x8 new 0xd
CPU Reset (Triple Fault)
EIP=0000000000101b2a RSP=00000000000fffd8

Root Cause: The page fault handler (vector 0xe) triggered because RSP exceeded the stack page. When pushing the exception stack frame, another page fault occurred, converting to a double fault (vector 0x8). Because the double fault handler was set to IST index 0 without a loaded TSS, the CPU attempted to push onto the same broken RSP, triggering a General Protection Fault (vector 0xd) that escalated into a triple fault reset.
Fix: Correctly wired the TSS descriptor into the GDT and loaded the Task Register via load_tss. Vector 8 was then explicitly bound to IST 0 in src/interrupts.rs (L59).

5. Concrete Verification & Test Traces

The stack overflow protection is verified by an automated integration test in tests/stack_overflow.rs:

cargo test --test stack_overflow

The test deliberately invokes an infinite recursive function:

#[allow(unconditional_recursion)]
fn stack_overflow() {
    stack_overflow(); // recursive call
    volatile::Volatile::new(0).read(); // prevent tail-call optimization
}

Observed Test Output:

test stack_overflow::stack_overflow_protection ...
[ok]
Execution completed without triple fault reset!

The test verifies that the double fault handler catches the overflow on the IST emergency stack and exits QEMU cleanly with an exit code of QemuExitCode::Success.

6. Interview Questions & Tactical Answers

Q1: "Why do we need a GDT in 64-bit mode if segmentation is dead?"

Tactical Answer: "While 64-bit mode flattens memory segmentation by forcing base addresses of code, data, and stack segments to 0 and ignoring limit checks, the CPU still uses segment selectors for privilege rings (ring 0 vs ring 3). Furthermore, loading the Task Register (TR) with ltr requires a valid 16-byte TSS descriptor inside the GDT. Without a GDT, the CPU cannot switch privilege levels or utilize Interrupt Stack Tables."

Q2: "What is the difference between an interrupt stack frame and a regular function stack frame?"

Tactical Answer: "A standard function call pushes only the return address (RIP) via the call instruction. When an interrupt occurs, the hardware automatically pushes a 5-word InterruptStackFrame: the old Stack Segment (SS), Stack Pointer (RSP), CPU Flags (RFLAGS), Code Segment (CS), and Instruction Pointer (RIP). If the interrupt causes a privilege level change or is bound to an IST index, the CPU switches RSP to the new stack *before* pushing these five values."