Diagnosing ARM64 Boot Failures Under QEMU

Welcome back. You now have the critical userspace side of a bootable system: a static ARM64 BusyBox root filesystem with /init, initial device nodes, and a startup policy. This lesson turns that filesystem into a repeatable QEMU boot test and, more importantly, a diagnostic exercise.

We will boot the system on QEMU’s ARM virt machine, which provides an ARM PL011 UART exposed to Linux as ttyAMA0. QEMU deliberately bypasses Boot ROM and U-Boot here: it loads the kernel and initramfs directly. That makes the boundary between kernel bring-up, console routing, and first userspace execution visible without board-specific variables.

By the end, you should be able to interpret the boot transcript and distinguish four failure classes that are easy to confuse:

1. no working serial console;
2. missing or non-executable init;
3. a missing ELF dynamic loader;
4. a missing shared library after the loader has started.

---

Establish the QEMU boot contract

Use the static BusyBox archive created in the previous lesson. Keep the kernel, root filesystem, command line, and captured logs together as private implementation evidence.

For the QEMU virt machine, the kernel needs a few built-in capabilities. An initramfs cannot depend on modules, because modules live in the filesystem that has not yet reached userspace.

Capability	Kernel configuration intent	Why it matters
Initramfs support	CONFIG_BLK_DEV_INITRD=y	Accepts the archive supplied through QEMU’s -initrd option.
gzip decompression	CONFIG_RD_GZIP=y	Decompresses rootfs.cpio.gz.
PL011 UART driver	CONFIG_SERIAL_AMBA_PL011=y	Supports QEMU virt’s serial device.
PL011 console	CONFIG_SERIAL_AMBA_PL011_CONSOLE=y	Allows kernel messages on ttyAMA0.
Device filesystem	CONFIG_DEVTMPFS=y	Allows /dev to be populated dynamically after startup.

If you have the kernel build directory, inspect its final configuration rather than assuming these features were selected:

export KERNEL_BUILD=/absolute/path/to/linux-build

grep -E 'CONFIG_(BLK_DEV_INITRD|RD_GZIP|SERIAL_AMBA_PL011|SERIAL_AMBA_PL011_CONSOLE|DEVTMPFS)=' \
    "$KERNEL_BUILD/.config"

A line ending in =y means the feature is built into the image. If the configuration is unavailable, boot evidence is still useful, but an early failure becomes harder to attribute.

Set up stable paths for this lab:

cd ~/gateway-private

export LAB="$PWD/labs/05-qemu-arm64-initramfs"
export ROOTFS="$PWD/labs/04-busybox-rootfs/rootfs"
export OUT="$LAB/out"

# Point this at an ARM64 Linux Image built or supplied for QEMU virt.
export KERNEL_IMAGE=/absolute/path/to/arm64/Image

mkdir -p "$OUT"

test -f "$KERNEL_IMAGE"
test -f "$OUT/../rootfs.cpio.gz" 2>/dev/null || true
test -d "$ROOTFS"
command -v qemu-system-aarch64

If qemu-system-aarch64 is absent, install the QEMU system emulator package through APT, then verify the executable again:

sudo apt update
sudo apt install qemu-system-arm
command -v qemu-system-aarch64

On Ubuntu, the qemu-system-arm package provides both ARM and AArch64 system emulators.

Create a local copy of the known-good archive in this lab’s output directory:

cp "$PWD/labs/04-busybox-rootfs/out/rootfs.cpio.gz" \
   "$OUT/rootfs-good.cpio.gz"

ls -lh "$KERNEL_IMAGE" "$OUT/rootfs-good.cpio.gz"

The direct-boot command is:

qemu-system-aarch64 \
    -machine virt \
    -cpu cortex-a53 \
    -m 512M \
    -nographic \
    -no-reboot \
    -kernel "$KERNEL_IMAGE" \
    -initrd "$OUT/rootfs-good.cpio.gz" \
    -append "console=ttyAMA0,115200 rdinit=/init loglevel=7"

Interpret the important arguments carefully:

• -machine virt selects QEMU’s generic 64-bit Arm virtual platform.
• -kernel loads an uncompressed ARM64 kernel Image directly.
• -initrd places the compressed cpio archive in RAM so the kernel can unpack it.
• console=ttyAMA0,115200 routes kernel output to the PL011 UART model.
• rdinit=/init explicitly tells the kernel which initramfs program to execute.
• -nographic maps serial I/O into the current terminal instead of opening a graphical window.
• -no-reboot keeps a panic visible rather than immediately restarting the virtual machine.

To leave QEMU, press Ctrl-A, release it, then press X.

A successful boot should include your rcS messages:

rcS: mounting pseudo-filesystems
rcS: minimal BusyBox rootfs ready

You should then reach a BusyBox shell. Confirm that the kernel command line, pseudo-filesystems, and device nodes agree with the design:

cat /proc/cmdline
mount
ps
ls -l /dev/console /dev/ttyAMA0 /dev/null
echo "console path works" > /dev/console

The distinction between /dev/console and /dev/ttyAMA0 matters:

• /dev/ttyAMA0 is the particular hardware UART driver instance for this QEMU platform.
• /dev/console is the kernel’s abstract currently selected console endpoint.
• The console=ttyAMA0 command-line setting connects those two concepts for this boot.

The static /dev/console node in the archive is needed before userspace mounts devtmpfs. Afterwards, devtmpfs exposes the actual UART node, including /dev/ttyAMA0.

---

Read boot output as evidence, not decoration

A boot log is chronological evidence. Start diagnosis at the last trustworthy message, rather than searching only for the final panic. The practical question is: which execution boundary was crossed successfully?

QEMU terminal transport
Kernel serial console
Initramfs unpacked and mounted
Kernel executes /init
BusyBox init executes rcS
Shell starts on a terminal

The kernel documentation provides a compact order for investigating failed init execution, from root filesystem mounting through console setup, dependencies, architecture, and scripts.

Use this evidence table during the lab.

Last reliable evidence	Likely boundary reached	First checks
Absolutely no kernel text in the terminal	QEMU transport or kernel console is wrong	QEMU command, -nographic, console=, UART driver built in
Kernel messages appear, but no rcS: messages	Kernel started but did not reach usable /init	Initramfs supplied, /init archived, mode, ELF architecture, interpreter
rcS: messages appear, then a shell warning or no usable prompt	PID 1 ran; issue is terminal or init policy	/etc/inittab, /dev/ttyAMA0, devtmpfs, serial line
Shell says an executable is “not found,” while ls shows it exists	The shell ran, but the ELF interpreter may be absent	readelf -lW, inspect PT_INTERP path
Loader reports error while loading shared libraries	Dynamic loader ran successfully	readelf -dW, target library locations and SONAMEs

The Bootlin QEMU labs show the same diagnostic progression: first a missing init, then a misleading “not found” error caused by a missing interpreter rather than a missing executable.

---

Failure class 1: a missing console is not a failed boot

First, provoke a console failure without modifying your root filesystem. Replace ttyAMA0 with ttyS0:

qemu-system-aarch64 \
    -machine virt \
    -cpu cortex-a53 \
    -m 512M \
    -nographic \
    -no-reboot \
    -kernel "$KERNEL_IMAGE" \
    -initrd "$OUT/rootfs-good.cpio.gz" \
    -append "console=ttyS0,115200 rdinit=/init loglevel=7"

On the QEMU ARM virt platform, ttyS0 is not the PL011 UART console. With -nographic, you may see no useful output at all, despite QEMU starting and the kernel potentially executing.

This is a common bring-up trap: silence does not establish that the CPU, kernel, or root filesystem failed. It establishes only that you have no trusted observation channel.

Restore the correct setting:

console=ttyAMA0,115200

For a real AM62x board later in the course, the equivalent reasoning stays the same, but the UART’s device name, clock setup, pin multiplexing, and Device Tree status will be board-specific. Never copy a QEMU console name into a physical board configuration.

The job-control warning is a different issue

Suppose the system prints:

/bin/sh: can't access tty; job control turned off

This is not a missing console. You have kernel output and BusyBox reached a shell. The warning means BusyBox init started the shell through the generic /dev/console endpoint, which is sufficient for basic I/O but may not supply the shell with a controlling terminal.

After devtmpfs is mounted, make the terminal explicit in /etc/inittab:

ttyAMA0::respawn:-/bin/sh

This refers to /dev/ttyAMA0. Recreate the archive after the edit, then boot it again. A change like this belongs in your private repository as a controlled configuration commit, accompanied by a before-and-after boot log.

---

Failure class 2: missing or broken /init

The kernel cannot continue normally without a first userspace process. A kernel message may include text such as:

Failed to execute /init (error -2)
Kernel panic - not syncing: No working init found.

Error -2 corresponds to ENOENT, usually “no such file or directory.” However, it is not proof that /init itself is absent. The same error can result when /init is an ELF executable whose dynamic loader is missing, or when /init is a script whose shebang interpreter is missing.

Before changing anything, inspect the archive on the host:

gzip -dc "$OUT/rootfs-good.cpio.gz" \
    | cpio -itv \
    | grep -E '(\./init|\./sbin/init|\./bin/sh|\./bin/busybox)$'

For the static design from the previous lesson, you should find:

• ./init as a link to bin/busybox;
• ./bin/busybox as an ARM64 static executable;
• BusyBox applet links such as ./sbin/init and ./bin/sh.

Also inspect the original filesystem directory:

ls -l "$ROOTFS/init" "$ROOTFS/bin/busybox" "$ROOTFS/sbin/init" "$ROOTFS/bin/sh"

"${TARGET_PREFIX}file" "$ROOTFS/bin/busybox"

"${TARGET_PREFIX}readelf" -lW "$ROOTFS/bin/busybox" \
    | grep 'Requesting program interpreter' || true

The static BusyBox binary should not print a requested program interpreter. If it does, you are not testing the intentionally static bootstrap design.

To create a safe missing-init test, make a root filesystem copy using privileged copy operations so the static character-device nodes remain device nodes:

export NOINIT="$LAB/rootfs-no-init"

sudo rm -rf "$NOINIT"
sudo cp -a "$ROOTFS" "$NOINIT"

sudo rm -f \
    "$NOINIT/init" \
    "$NOINIT/sbin/init" \
    "$NOINIT/bin/init" \
    "$NOINIT/bin/sh" \
    "$NOINIT/linuxrc"

Removing the fallback candidates is deliberate. Without it, the kernel might fail to execute /init but successfully try another BusyBox init candidate, concealing the intended failure.

Package this controlled variant:

(
    cd "$NOINIT"
    sudo find . -print0 \
        | LC_ALL=C sort -z \
        | sudo cpio --null -o --format=newc --owner=0:0
) > "$OUT/rootfs-no-init.cpio"

gzip -n -9 -c "$OUT/rootfs-no-init.cpio" > "$OUT/rootfs-no-init.cpio.gz"

Boot it using the same QEMU command, replacing only the -initrd argument. Record the last successful kernel line and the attempted init path. Then restore the good archive; do not “fix” this test case in place.

A direct inspection checklist for a genuine /init problem is:

# Does the archive contain the intended path?
gzip -dc rootfs.cpio.gz | cpio -it | grep '^init$'

# Is it executable or a valid symlink?
ls -l rootfs/init

# If it is a script, is its interpreter present and executable?
head -n1 rootfs/init

# If it is ELF, is it ARM64 and is its loader available?
"${TARGET_PREFIX}file" rootfs/init
"${TARGET_PREFIX}readelf" -lW rootfs/init

A script with #!/bin/sh requires /bin/sh to work. A dynamically linked binary requires its ELF interpreter to exist at the exact path embedded in the program.

---

Failure class 3: the executable exists, but the dynamic loader does not

The static BusyBox bootstrap is intentionally robust, but normal applications are often dynamically linked. Build a small dynamic test program without replacing /init; this preserves a working shell for observation.

cat > "$LAB/hello-dyn.c" <<'EOF'
#include <stdio.h>

int main(void)
{
    puts("hello from dynamically linked ARM64 userspace");
    return 0;
}
EOF

"${TARGET_PREFIX}gcc" -O2 \
    -o "$LAB/hello-dyn" \
    "$LAB/hello-dyn.c"

"${TARGET_PREFIX}file" "$LAB/hello-dyn"

"${TARGET_PREFIX}readelf" -lW "$LAB/hello-dyn" \
    | grep 'Requesting program interpreter'

"${TARGET_PREFIX}readelf" -dW "$LAB/hello-dyn" \
    | grep NEEDED

Do not use Ubuntu’s native file, readelf, or ldd as the decisive runtime check. The cross-toolchain’s inspection utilities identify the ARM64 file accurately, while ldd on the x86-64 host may be unsuitable or unsafe for target binaries.

Install the executable in the root filesystem but deliberately do not copy its loader or libraries:

install -m 0755 "$LAB/hello-dyn" "$ROOTFS/bin/hello-dyn"

(
    cd "$ROOTFS"
    find . -print0 \
        | LC_ALL=C sort -z \
        | cpio --null -o --format=newc --owner=0:0
) > "$OUT/rootfs-loader-missing.cpio"

gzip -n -9 -c "$OUT/rootfs-loader-missing.cpio" \
    > "$OUT/rootfs-loader-missing.cpio.gz"

Boot rootfs-loader-missing.cpio.gz. At the BusyBox shell:

ls -l /bin/hello-dyn
/bin/hello-dyn

The file should be visible. Yet the execution attempt may report:

/bin/hello-dyn: not found

That message is misleading because the shell asks the kernel to execute /bin/hello-dyn, the kernel finds it, then the kernel cannot find the interpreter named in its ELF PT_INTERP segment.

For a typical glibc ARM64 toolchain, the interpreter is often:

/lib/ld-linux-aarch64.so.1

For musl it is commonly an architecture-specific musl loader such as:

/lib/ld-musl-aarch64.so.1

These are examples, not paths to guess. Your readelf output is authoritative.

Find the actual SDK sysroot and locate the required interpreter:

export SYSROOT="$("${TARGET_PREFIX}gcc" -print-sysroot)"

printf 'SDK sysroot: %s\n' "$SYSROOT"

find "$SYSROOT" -type f -o -type l | grep 'ld-linux\|ld-musl'

Copy the loader to the exact PT_INTERP pathname reported by your executable. For example, if readelf reported /lib/ld-linux-aarch64.so.1, the target filesystem must contain that exact path:

# Replace these values with paths established by your inspection.
export INTERPRETER=/lib/ld-linux-aarch64.so.1
export LOADER_SOURCE="$SYSROOT/lib/ld-linux-aarch64.so.1"

install -d "$ROOTFS$(dirname "$INTERPRETER")"
cp -aL "$LOADER_SOURCE" "$ROOTFS$INTERPRETER"

-aL copies the loader’s resolved file content. This avoids leaving a loader symlink that points to an absent target within the initramfs.

Rebuild the archive, boot again, and rerun /bin/hello-dyn. You have now moved the failure boundary: the kernel found the executable and its interpreter, so the dynamic loader can begin resolving DT_NEEDED library entries.

---

Failure class 4: the loader runs, but a shared library is absent

Once the interpreter exists, an error commonly changes to something like:

/bin/hello-dyn: error while loading shared libraries: libc.so.6:
cannot open shared object file: No such file or directory

This is progress. The loader itself ran, parsed the dynamic section, and could name the missing dependency.

Use the host-side ELF metadata to enumerate requirements:

"${TARGET_PREFIX}readelf" -dW "$LAB/hello-dyn" \
    | grep NEEDED

A dynamically linked ELF program has two distinct runtime contracts:

ELF field	Question it answers	Typical failure if absent
PT_INTERP program-header entry	Which loader must the kernel execute first?	An existing executable reports “not found.”
DT_NEEDED dynamic-section entries	Which shared objects must the loader locate?	Loader reports a named missing .so file.

For the simple glibc test, libc.so.6 is usually the important dependency. Locate the exact library in the sysroot:

find "$SYSROOT" -name 'libc.so.6' -ls

Copy it into a standard target library directory, preserving the filename expected by the loader:

install -d "$ROOTFS/lib"

# Replace the source path with the actual result from the previous find command.
cp -aL "$SYSROOT/lib/libc.so.6" "$ROOTFS/lib/libc.so.6"

Rebuild the archive and test once more. If the loader reports another library, repeat the same evidence-driven process: read DT_NEEDED, locate the target library in the SDK sysroot, and place it in an appropriate target library directory.

There is one important C-library nuance:

• With glibc, the loader and libc.so.6 are usually distinct files, so you often see a loader failure first and a libc failure second.
• With musl, the dynamic linker commonly also provides the C library. Copying the correct musl loader may therefore resolve a trivial C program immediately.
• A C++ application, a program using threads, TLS, compression, crypto, or vendor SDK libraries normally has a wider dependency closure. Inspect it; do not copy libraries based on habit.

In a production Yocto image, the package manager and recipe metadata construct this runtime closure. This manual experiment is valuable precisely because it exposes what the build system later automates: interpreter placement, library ownership, SONAME-compatible files, and runtime dependency declarations.

---

A compact diagnosis procedure

When QEMU fails, use this order before changing files:

1. Confirm the observation channel.
   Check QEMU uses -nographic and the kernel command line uses console=ttyAMA0,115200 for virt.

2. Identify the last successful stage.
   Kernel text establishes a console. rcS text establishes that /init and BusyBox init ran. A shell prompt establishes that startup policy worked.

3. Inspect the archive, not just the source directory.
   Use gzip -dc ... | cpio -itv to verify /init, mode bits, links, and device nodes in the actual artifact QEMU receives.

4. Classify an ELF executable.
   Use file for architecture and static-versus-dynamic status, readelf -lW for PT_INTERP, and readelf -dW for DT_NEEDED.

5. Fix one missing contract at a time.
   Add the loader before chasing libraries. Add the reported library before investigating application logic. Repackage, reboot, and preserve the resulting log.

For an automotive or industrial gateway, this approach scales directly to serial bring-up on real hardware. The names change, and the evidence may arrive through an FTDI adapter rather than QEMU, but the engineering discipline is identical: make the boot artifacts reproducible, capture the whole transcript, and locate the precise failed contract.

---

Key takeaways

A successful minimal QEMU boot demonstrates a complete kernel-to-userspace handoff:

• QEMU’s ARM virt machine uses the PL011 UART, exposed as ttyAMA0; console=ttyAMA0,115200 is the relevant kernel setting.
• Silence is primarily a console-observability problem, not immediate proof of a kernel or root-filesystem failure.
• /init must be present, executable, architecture-compatible, and able to run all of its own interpreters or libraries.
• “Not found” for an executable that ls can see is strong evidence of a missing ELF interpreter.
• PT_INTERP identifies the dynamic loader required by the kernel; DT_NEEDED identifies shared objects required by that loader.
• A static BusyBox bootstrap deliberately avoids the loader and library dependency chain, making it an effective first bring-up baseline.
• Archive inspection and complete serial logs are first-class engineering artifacts, not incidental debug output.

Next, you will move below the QEMU direct-boot shortcut and construct an ARM boot-artifact and address map from Boot ROM through SPL or TPL, U-Boot Proper, kernel, root filesystem, and finally PID 1.