Localizing Early-Boot Failures from Serial Logs

Welcome to the boot-failure investigation module. In a production gateway, the serial log is often the first and most valuable piece of evidence after a failed update, a board revision, a device-tree change, or a new storage layout. The goal is not merely to recognize error strings: it is to identify the last stage that demonstrably succeeded, separate that from missing observability, and form the smallest credible set of suspects.

This lesson builds a practical classification method for failures spanning the bootloader, device tree, kernel, initramfs, root filesystem mount, console, and PID 1. Although the eventual hardware target is AM62x, the diagnostic method applies to most ARM Linux boards.

---

Treat serial output as evidence, not as a diagnosis

A serial boot log is a timeline emitted by several independent programs. Each program owns a distinct portion of initialization and normally leaves recognizable textual fingerprints:

1. Early firmware and boot stages establish clocks, power, SRAM, DRAM, and secure or privileged execution services.
2. U-Boot discovers boot media, loads artifacts, constructs kernel arguments, and transfers control.
3. Linux early kernel initializes CPU state, memory management, interrupts, and the driver model.
4. Kernel storage and VFS code finds and mounts the intended root filesystem.
5. PID 1 begins userspace policy, commonly through BusyBox init or systemd.

The fundamental rule is:

Localize a fault immediately after the last proven-successful boundary, but do not mistake a missing serial console for proof that the processor stopped.

For example, this does not prove that Linux never started:

Starting kernel ...

It could mean the kernel crashed during its earliest instructions. It could also mean that the kernel is running but its normal console has been incorrectly configured or its UART driver has not yet probed. Treat it as a handoff, device-tree, early-kernel, or console-observability candidate set until further evidence narrows it.

Before changing software, preserve one complete raw log. Your incident record should include:

• Board type and revision, boot medium, and whether the reset was a cold power cycle or warm reset.
• UART adapter, voltage level, port, and terminal configuration.
• Exact images and image hashes where available.
• The last expected line and first unexpected line.
• What changed since the most recent known-good boot: bootloader environment, DTB, kernel, rootfs, storage image, or hardware.

This discipline prevents a common bring-up failure mode: repeatedly rebuilding several artifacts, thereby destroying the information needed to identify the original fault.

---

The AM62x stage map: locate the last confirmed owner

On AM62x-class systems, boot contains more intermediate firmware than a small microcontroller boot. The exact artifacts and banners depend on the SDK version and security configuration, but the useful ownership boundaries are consistent:

Last evidence seen in the log	What has been proven	Primary fault region
No output at all	Nothing software-related has been proven	Power, reset, boot straps, boot media, ROM behavior, UART wiring/configuration
Early SPL or firmware banner, but no U-Boot banner	ROM found an initial image and early code began	Initial boot artifact, DDR bring-up, firmware handoff, authentication, boot media
U-Boot banner or prompt	DRAM is usually usable and U-Boot Proper is executing	U-Boot configuration, environment, storage/network loading, boot artifact selection
Starting kernel ..., but no Linux messages	U-Boot accepted the boot command and initiated handoff	Kernel image format, load addresses, DTB validity, ARM64 handoff, early console
Linux version and early kernel messages	Kernel entry and early initialization succeeded	Kernel configuration, device tree, console transition, drivers
Root-device discovery followed by VFS mount failure	Kernel and relevant portions of storage support run	root= argument, DT storage description, driver availability, partition/filesystem
VFS: Mounted root ..., then no init or panic	Kernel mounted a root filesystem	Initramfs handover or PID 1 program, dynamic loader, file permissions, rootfs contents
Login prompt or service logs	Kernel reached userspace	Init configuration or later userspace service behavior, outside early boot

This is a localization tool, not a guarantee that a component is flawless. For example, U-Boot may successfully load an incorrect DTB; it has succeeded as a bootloader but handed the kernel incorrect hardware metadata.

The TI AM62x documentation gives a useful reference sequence: PMIC power delivery, Boot ROM, R5 SPL, trusted firmware components, U-Boot Proper, kernel, and userspace.

A normal multi-stage example from Bootlin illustrates how visually different banners delimit stages: BL2, BL31, OP-TEE, then U-Boot. That specific log is from an STM32 platform, not AM62x, but the method transfers directly: every banner marks a component that has already executed successfully.

---

A repeatable serial-log triage method

When a board fails, use this order of reasoning.

1. Find the last positive assertion

A positive assertion is a line that proves a concrete event occurred:

• U-Boot ... proves U-Boot Proper reached its banner.
• DRAM: 2 GiB proves U-Boot’s DRAM initialization and reporting reached that point.
• Loading Environment from MMC... OK proves a particular environment read succeeded.
• Starting kernel ... proves U-Boot chose an image and attempted the Linux handoff.
• Linux version ... proves kernel control flow began.
• VFS: Mounted root ... proves the root filesystem mount succeeded.
• Run /sbin/init as init process or systemd startup messages prove userspace has started.

Lines such as “trying,” “loading,” or “probing” are weaker. They describe an attempted action, not necessarily a completed one.

2. Identify the next failed contract

Each boundary has a contract:

Boundary	Contract that must hold	Typical evidence when it fails
Boot ROM to first loader	Correct boot source and readable/authentic initial image	No output, ROM-specific status, or an early firmware failure
SPL to U-Boot	DRAM initialized; next-stage artifact readable and valid	SPL banner appears, U-Boot banner does not
U-Boot to kernel	Correct kernel image, valid FDT, non-overlapping addresses, valid boot arguments	Load failure, bad image/FDT error, silence after Starting kernel ...
Kernel to root filesystem	Required controller and filesystem driver available; root= identifies the intended partition	Unable to mount root fs, unknown-block, device timeout
Initramfs to real root	/init works and mounts or switches to the intended root	Failed to execute /init, shell/script errors, root-switch failure
Root filesystem to PID 1	An executable init program and its dependencies exist	No working init found, Requested init ... failed

3. Separate a data-plane failure from a visibility failure

A wrong console= parameter is especially deceptive. Linux can be healthy enough to mount the root filesystem while serial output is absent because:

• The kernel command line names the wrong UART.
• The UART node is disabled or has an incorrect pinmux/clock in the DTB.
• The selected baud rate differs from the terminal.
• Early output used earlycon, but the normal console did not take over.
• The terminal adapter or its electrical connection is wrong.

Therefore, a silent gap after the bootloader should be investigated with two competing hypotheses:

• Execution hypothesis: the kernel did not continue.
• Console hypothesis: the kernel continued, but output moved or disappeared.

Do not respond by randomly changing the kernel, rootfs, and serial settings together. Restore a known-good console string from the board’s reference boot arguments, then change only one dimension at a time.

4. Verify what U-Boot actually loaded

When U-Boot is available, it is the best place to check assumptions before touching the kernel. Useful inspection commands vary with the configured features, but the general categories are:

• printenv bootargs bootcmd to inspect the kernel command line and boot script.
• bdinfo to inspect DRAM boundaries and load-address context.
• mmc list, mmc part, and filesystem listing/loading commands to validate the intended storage path.
• fdt addr and fdt print /chosen to inspect the in-memory device tree and its /chosen node.
• iminfo to inspect an image container such as FIT, where supported.

For diagnosis, ask concrete questions:

• Did the intended kernel and DTB load from the intended partition?
• Are the kernel, DTB, and initramfs at distinct addresses within DRAM?
• Does bootargs contain the expected console and root= values?
• Does the in-memory FDT match the board revision and peripheral topology?
• Is U-Boot booting a stale environment, alternate boot target, or fallback artifact?

The Linux Foundation’s U-Boot overview shows why bootargs, bootcmd, loaded artifacts, and the selected boot command belong in the same investigation.

---

Distinguishing device-tree, kernel, and console failures

These three failures can occur in the same region of the log, but their evidence differs.

Device tree failure

The DTB tells Linux what hardware exists, how it is addressed, and which clocks, interrupts, pinmux settings, and dependencies it needs. A wrong DTB can produce several symptom classes:

• U-Boot may reject an invalid FDT before it starts Linux.
• Linux may start but print an incorrect Machine model.
• The UART may not become a working console.
• An MMC, Ethernet, or SPI controller may fail to probe, making the required root device absent.
• Drivers may emit probe failures involving clocks, regulators, interrupts, or deferred probing.

A storage-root failure can therefore be caused by a device tree, even though the visible panic is a VFS mount failure. The immediate localization is “root mount”; the next causal investigation includes both the storage driver and the DT node describing that controller.

A robust confirmation method is to inspect the DTB in U-Boot and compare its board model, /chosen properties, and relevant controller node against the artifact you intended to deploy. A filename in a boot script is not proof that the expected binary was loaded.

Kernel failure

If the Linux version banner and early messages appear, the kernel is executing. At that point, a failure can involve:

• A kernel configuration missing a required driver.
• An incompatibility between kernel and DTB.
• An early exception or panic.
• Memory corruption from invalid reserved-memory information or artifact overlap.
• A driver fault during probe.

The key distinction is that kernel logs prove control reached Linux. Do not continue debugging U-Boot as the primary suspect unless the evidence points back to the artifacts or arguments it supplied.

Console failure

A console failure often manifests as a sharp cutoff at the handoff boundary. It may also appear later, after some early messages. Look for evidence from another source before declaring a dead kernel:

• Ethernet link and network reachability if a known image starts networking automatically.
• LEDs or watchdog behavior already characterized in a known-good build.
• Output from a valid early console configuration.
• A serial log from a known-good kernel using the same physical connection.

For early bring-up, keeping serial console configuration stable while changing other boot components reduces ambiguity.

---

Initramfs, root mount, and PID 1 are different failures

These late-boot terms are often conflated because all of them happen after the kernel banner. They are separate contracts.

Initramfs

An initramfs is an archive unpacked into memory by the kernel. If it contains /init, that program becomes the first userspace program. It may load essential modules, decrypt storage, choose an A/B slot, mount a persistent filesystem, and then switch to it.

Evidence such as these points to the initramfs stage:

Trying to unpack rootfs image as initramfs...
Failed to execute /init

Likely causes include a missing /init, a non-executable script, an invalid shebang, a missing interpreter, or an architecture mismatch. If /init is a dynamically linked executable, its ELF interpreter and shared libraries must also exist in the initramfs.

Root filesystem mount

A persistent root filesystem is commonly selected through root= in bootargs. The kernel must have all of the following:

• The correct storage controller driver.
• A usable description of that controller, usually from the DTB.
• Partition discovery support.
• The filesystem driver.
• The correct root device identifier and sufficient time for the device to appear.

Typical evidence is:

VFS: Cannot open root device "..." or unknown-block(...)
Kernel panic - not syncing: VFS: Unable to mount root fs

This proves the kernel ran far enough to attempt a mount. The investigation should start with root=, partition identity, storage-driver logs, and the DTB, not with application code.

PID 1

Once the kernel reports a successful root mount, it tries to run an init program. If no explicit init= parameter was supplied, the kernel tries conventional init paths. Failure here commonly produces:

VFS: Mounted root ... 
Kernel panic - not syncing: No working init found.
Try passing init= option to kernel.

That is not a root-mount failure. The root filesystem exists and was mounted; its contents cannot provide a runnable init program.

The Bootlin lab makes the distinction visible: it first shows a successful root mount, then a missing /dev issue, and finally the “No working init found” panic.

A minimal BusyBox system can reach PID 1 and still appear “dead” if its inittab does not start a shell on the correct console, or if its startup script lacks execute permission. That is a userspace configuration failure after kernel boot, not a kernel failure.

---

Worked log classifications

Consider these abbreviated cases.

Case A: U-Boot cannot find the kernel

U-Boot 2024.01
MMC: mmc@...: 0
Scanning mmc 0:1...
** Unable to read file Image **

Localization: U-Boot boot-media or boot-script region.

Reasoning: U-Boot runs and MMC is enumerated. There is no evidence that the kernel was loaded or started. Check the selected boot target, partition, filesystem, filename, and environment before examining the kernel.

Case B: Kernel runs, but cannot mount root

Starting kernel ...

[    0.000000] Linux version ...
[    1.842000] mmc0: new HS200 MMC card
[    2.104000] VFS: Cannot open root device "mmcblk0p2" or unknown-block(0,0)
[    2.115000] Kernel panic - not syncing: VFS: Unable to mount root fs

Localization: root filesystem mount.

Reasoning: Linux started and the MMC device was discovered. The likely causes are an incorrect root= identifier, a partition-layout mismatch, missing filesystem support, or a rootwait/timing issue. If MMC discovery did not occur, widen the investigation to the DTB and MMC-driver configuration.

Case C: Root mounted, but PID 1 fails

[    1.900000] VFS: Mounted root (ext4 filesystem) readonly
[    1.930000] Run /sbin/init as init process
[    1.940000] Kernel panic - not syncing: No working init found.

Localization: PID 1 and rootfs content.

Reasoning: The kernel successfully found and mounted the root filesystem. Inspect the filesystem for /sbin/init, its mode bits, architecture, ELF interpreter, linked libraries, and any explicit init= command-line override. Reformatting the boot partition or changing the DTB is not the first response.

---

An operational checklist for the AM62x gateway lab

For each failed boot, produce a short conclusion in this format:

Field	Example
Last confirmed stage	U-Boot Proper loaded from eMMC
First failed contract	Kernel could not mount intended root partition
Evidence	Linux banner present; MMC driver logs present; VFS reports unknown root device
Leading hypotheses	Wrong root= value; stale partition layout; missing ext4 support
One discriminating check	Compare bootargs with actual partition table and inspect kernel storage logs
Change made	None yet; evidence preserved
Result	Pending

This format distinguishes an observation from an assumption and makes handover between engineers much more efficient. It is also suitable for sanitized portfolio evidence later: publish the stage map and anonymized log interpretation, but keep device-specific credentials, internal addresses, and security-sensitive artifacts private.

---

The essential takeaways are:

• Classify boot failures by the last component that proved it executed, not by the first scary-looking warning.
• A silent serial log is an observability problem until evidence shows otherwise.
• U-Boot success, kernel entry, root mount, initramfs execution, and PID 1 execution are separate boundaries.
• A VFS root-mount failure may originate in boot arguments, storage drivers, or a device tree, while No working init found means the root mount already succeeded.
• Preserve the original log and test one discriminating hypothesis at a time.

Next, you will examine why manually assembled root filesystems frequently fail in ways that are difficult to reproduce: host contamination, omitted runtime dependencies, timestamp variation, and long-term maintenance drift.