Tracing an x86-64 Linux System Call: From User Space to Kernel Return

Welcome. This course approaches complex software from its lowest enforceable boundaries outward: CPU privilege transitions, memory isolation, user processes, IPC, and eventually browser rendering and sandboxing. In this first module, the focus is the x86-64 boundary between ordinary application code and kernel code.

This lesson traces one concrete Linux system call, write(1, "hello\n", 6), from its userspace setup through the processor’s syscall instruction, Linux’s entry code and dispatch, and the return to the instruction after syscall. By the end, you should be able to narrate the path while distinguishing what the CPU hardware does from what Linux software must do.

---

Start with the contract: write from a userspace program

A system call is the controlled interface through which a ring-3 program requests work that requires kernel authority: reading a file, creating a process, mapping memory, communicating over a socket, and so on. It is not an ordinary function call. An ordinary call can jump only to code that the current process is permitted to execute; a system call causes a processor-defined transition into ring 0 at an entry address chosen earlier by the kernel.

At the source level, a C program might say:

write(1, "hello\n", 6);

The familiar C library function is normally a wrapper. It may perform checking, cancellation handling, or error translation, but its essential low-level job is to place a syscall number and arguments in the Linux x86-64 syscall ABI registers, then execute the syscall instruction.

For the 64-bit Linux syscall ABI:

Register	At entry to the kernel
rax	System-call number
rdi	Argument 1
rsi	Argument 2
rdx	Argument 3
r10	Argument 4
r8	Argument 5
r9	Argument 6
rcx	Will be overwritten with userspace return address
r11	Will be overwritten with saved flags

For write on x86-64 Linux, the syscall number is $1$. A minimal assembly-level setup in GNU assembler’s AT&T syntax looks like this:

mov $1, %eax             # __NR_write
mov $1, %edi             # file descriptor: stdout
lea message(%rip), %rsi  # address of "hello\n"
mov $6, %edx             # byte count
syscall

Using the 32-bit register names such as eax and edi is common here: writing one of them zero-extends the corresponding 64-bit register. The string pointer is still a full 64-bit virtual address.

The fourth argument deserves special attention. The ordinary System V x86-64 C calling convention uses rcx as its fourth integer argument register. The syscall convention instead uses r10, because the CPU itself overwrites rcx during syscall. Linux’s internal syscall wrappers can rearrange values as necessary before entering ordinary C-compatible code.

The supplied “How a Linux system call works” diagram gives a useful role-level map of this journey. One correction is important before relying on it: on modern x86-64, the syscall instruction is not an old-style software interrupt and does not automatically push a frame or switch stacks. Linux explicitly builds the saved state and switches to a kernel stack in its entry code.

---

What the processor does at syscall

The instruction syscall is a deliberately narrow gate. During boot, Linux programs model-specific registers, or MSRs, on each CPU. The important ones here are:

• IA32_LSTAR: the 64-bit kernel entry address used by syscall.
• IA32_STAR: supplies the code and stack segment selectors used on entry and return.
• IA32_FMASK: specifies flag bits to clear as the kernel begins execution.

When ring-3 code executes syscall, the CPU performs a transition with a tightly specified register-level effect:

1. It saves the address of the next userspace instruction in rcx.
2. It saves the userspace rflags value in r11, then masks selected active flags for kernel execution.
3. It loads kernel privilege segment state and loads rip from IA32_LSTAR.
4. It does not push anything on a stack and does not change rsp.

That final point explains much of the entry assembly. At the first kernel instruction, rsp still points at a userspace stack, whose contents and address are untrusted. Kernel code must not treat it as a safe execution stack.

The modern entry routine begins at entry_SYSCALL_64. Its first responsibilities can be understood in terms of trust and location rather than memorizing every macro:

1. Establish kernel per-CPU state. swapgs exchanges the active GS-base value with a kernel-controlled one. Linux uses the kernel GS base for per-CPU data, including information needed to find a safe stack. swapgs changes a base register; it is not itself a stack switch.

2. Preserve the userspace stack pointer. Linux saves the old rsp in per-CPU scratch storage. It needs that exact value later to resume the caller.

3. Switch to the current task’s kernel stack. Linux loads a known kernel stack top into rsp. On systems using page-table isolation, entry code may also switch to an appropriate kernel page-table context. These mitigation details evolve across kernel versions, but the essential rule does not: privileged execution proceeds on kernel-controlled state.

4. Construct a pt_regs record on that kernel stack. Linux pushes a software-built representation of the interrupted userspace context: userspace stack selector and pointer, flags, code selector, instruction pointer, original syscall number, and general-purpose-register state.

The source calls this struct pt_regs. It is a central object: tracing, signal delivery, ptrace, auditing, syscall filtering, and the return path all need a consistent representation of the task’s saved machine state. The exact layout will be the subject of the next lesson; for now, notice its origin. For a syscall, much of this record is constructed by Linux, rather than automatically pushed by the processor.

There is also a boundary of responsibility here. The kernel has gained authority, but the arguments in rdi, rsi, and other registers are still supplied by an untrusted process. In write, the buffer pointer identifies userspace memory, not kernel memory. Correct kernel code must validate and safely copy or access user-provided data; entering ring 0 does not make user pointers trustworthy.

---

From syscall number to Linux service routine

Once the entry code has captured the context, the current Linux source calls do_syscall_64. It receives a pointer to the saved register state and the syscall number. The dispatcher checks whether the number is valid for the active ABI, then selects the corresponding kernel syscall implementation.

Conceptually, dispatch is table lookup:

$$\text{syscall number}\mapsto \text{kernel handler}$$

For our example, the number $1$ selects the 64-bit write syscall implementation. The handler receives the semantic arguments:

• file descriptor $1$,
• pointer to "hello\n",
• byte count $6$.

From there, write follows kernel objects rather than raw hardware. Linux resolves file descriptor $1$ in the calling process’s descriptor table; it may refer to a terminal, a pipe, a redirected file, or something else. The syscall handler eventually returns a result. On success, that is normally the number of bytes written, here $6$.

At the raw syscall layer, errors are conventionally negative error values in rax, such as a negative value corresponding to a bad file descriptor or invalid address. A libc wrapper generally turns this into the C-facing convention of returning $-1$ and setting errno. That translation is why observing a syscall boundary and observing a C function boundary are not quite the same thing.

A useful way to validate the first half of the trace on a Linux machine is to run:

strace -e write ./your_program

strace shows the kernel-facing operation, not merely the source-level call spelling. If output is produced through printf, the trace may reveal one or more write calls, but buffering means the relationship is not necessarily one printf call per write syscall.

---

Returning safely to ring 3

When the selected handler finishes, it leaves its raw return value in rax, and Linux records that as the return value in the saved context. The kernel then performs exit work. Depending on what occurred during the call, this can include checking pending signals, rescheduling, tracing hooks, and security-related state transitions.

Linux has two principal architectural ways to return to userspace:

• sysretq is the fast return path for a clean, valid 64-bit syscall context.
• iretq is the more general return mechanism. Linux chooses it when the saved state requires a robust slow path, for example after state modifications that make the direct syscall-return assumptions unsafe.

For the ordinary fast sysretq case, the meaning matches the processor’s entry contract:

• Linux restores general-purpose registers from the saved context.
• It restores the saved userspace rsp.
• It places the saved userspace instruction pointer in rcx.
• It places the saved userspace flags in r11.
• It executes swapgs again to restore the userspace GS-base context.
• It executes sysretq.

The processor then resumes at the instruction immediately following syscall, in ring 3. From the caller’s viewpoint, the wrapper simply returns, with rax holding the result.

It is tempting to summarize the trace as “the program calls the kernel and the kernel returns.” That hides the important machinery. A more accurate compact account is:

1. A userspace wrapper places the syscall number and arguments in the syscall ABI registers.
2. syscall transfers control to Linux’s preconfigured entry address, while saving the return instruction pointer in rcx and flags in r11.
3. Linux replaces the unsafe userspace-stack context with kernel-controlled per-CPU state and a kernel stack, then builds pt_regs.
4. Linux validates and dispatches the numeric request to its syscall implementation.
5. The handler produces a result in rax; Linux performs required exit work.
6. Linux restores a safe userspace context and returns through sysretq when possible, otherwise iretq.

---

The key takeaways are that a syscall is both an ABI contract and a privilege transition; syscall itself does less automatic saving than many people expect; and Linux entry code is responsible for creating the safe, inspectable state needed to run kernel code and later resume the task. The syscall number selects a service, but it does not grant trust to its arguments.

Next, we will look closely at the privilege-transition state itself: what fields are saved, why pt_regs resembles an interrupt-return frame, and how saved rip, rsp, flags, and segment state determine where execution can safely resume.