4.6.1. pstore & ramoops

4.6.1.1. Introduction to pstore

pstore, short for persistent storage, is a kernel feature commonly used to implement a persistent storage mechanism in Linux. It allows saving debugging information, crash logs, or other critical kernel state data, which can be retained even after a system reboot.
Its main purpose is to provide a mechanism to preserve key information after system crashes or reboots, helping developers debug and analyze issues.

4.6.1.2. pstore Principle

pstore writes logs in real-time to uncached memory. Therefore, the data stored by pstore can fully reflect the kernel logs before reboot, making pstore often the most important source of debugging information when analyzing unexpected reboots.
On the X5 platform, when an abnormal reboot (e.g., panic, wdog, etc.) occurs, the system performs a WarmReset, meaning the memory is not powered down. After the next reboot, users can retrieve the abnormal logs via specific nodes.

4.6.1.3. pstore Types

pstore supports the following types of log information:

console

pstore registers a console, and all logs output to the serial console are synchronously written into a specified address managed by pstore. Thus, the logs are identical to the serial console output (subject to log level control), and this memory segment is managed via a ring buffer.
After an abnormal reboot, logs can be retrieved from /sys/fs/pstore/console-ramoops-*.

ramoops

ramoops refers to a feature that uses RAM to store oops information. It dumps kmsg during critical paths of system crashes (panic/oops) to preserve logs. Unlike console logs, ramoops is not controlled by log level, making it a valuable supplement to console logs.
After an abnormal reboot, logs can be retrieved from /sys/fs/pstore/dmesg-ramoops-*.

sched

sched logs are a logging mechanism used to analyze scheduling issues before system reboot. They preserve scheduling information from each CPU prior to reboot, which is very helpful in diagnosing issues such as wdog. This feature requires enabling CONFIG_SCHED_LOGGER.
After an abnormal reboot, logs can be retrieved from /sys/fs/pstore/sched-ramoops-*.

ftrace

The pstore ftrace feature can be used to trace soft/hard hang issues occurring before reboot, capturing function call traces.
This feature is disabled by default due to its significant impact on system load. It is not recommended for regular use; the sched feature is preferred instead.

4.6.1.4. Introduction to ramoops

ramoops is part of the pstore system.
ramoops saves kernel crash information, log data, or other critical data into a reserved memory region. This memory region is not cleared during system reboot (or at least remains readable during the reboot process).

4.6.1.5. ramoops Usage

Enable ramoops in Kernel

The following kernel configuration options must be enabled:

CONFIG_PSTORE=y
CONFIG_PSTORE_RAM=y

Configure ramoops Reserved Memory

Modify the DTS to set the memory address space and size for ramoops. Since this memory is not managed by the Linux kernel, it must be reserved.

ramoops-related settings are located under the reserved_memory node.

Below is the default DTS configuration for the X5 system:

ramoops@d5000000 {
    compatible = "ramoops";
    reg = <0x0 0xd5000000 0x0 0x40000>;
    console-size = <0x8000>;
    pmg-size = <0x8000>;
    ftrace-size = <0x8000>;
    sched-size  = <0x8000>;
    record-size = <0x4000>;
    ecc-size = <0x0>;
};

Trigger Kernel Panic in Kernel

The kernel can actively trigger a kernel panic via sysrq to verify if the pstore mechanism is functioning:

echo c > /proc/sysrq-trigger

Verify ramoops Functionality

After an abnormal reboot, on the X5 platform, kernel logs retrieved via pstore are saved to /userdata/log/pstore/:

root@buildroot:~# ls -al /userdata/log/pstore/X5_Pstore-0017-1970_01_01_00_00_03/
total 76
drwxr-xr-x 2 root root  4096 Oct 15  2024 .
drwxr-xr-x 3 root root  4096 Jan  1 00:00 ..
-r--r--r-- 1 root root  1391 Jan  1 00:00 console-ramoops-0
-r--r--r-- 1 root root 27248 Jan  1 00:00 dmesg-ramoops-0
-r--r--r-- 1 root root 33922 Jan  1 00:00 sched-ramoops-0