4.5.3. CPU-BPU-DDR Stress Test

4.5.3.1. Test Principle

The principle of the CPU-BPU-DDR stress test primarily involves evaluating the performance and stability of the CPU, BPU, and DDR under high-load conditions. Below are the fundamental principles of stress testing for these components:

1. CPU DDR Stress Test

Test Objective: Use the stressapptest tool to simulate a high-load environment, performing extensive computation, data processing, and memory operations to evaluate system performance under multi-threaded concurrent tasks.
Test Purpose: Verify the stability and performance of the CPU under prolonged high load, ensuring that both the CPU and DDR can operate normally under heavy loads without issues such as crashes, overheating, or performance degradation.
stressapptest CPU Stress Principle: CPU stress testing is mainly achieved by multi-threaded execution of compute-intensive tasks, including: compute-intensive operations, parallel multi-threaded execution, etc.
- Thread Creation: stressapptest uses pthread_create() to create multiple threads, each executing computational tasks. The number of threads is typically specified by the user via the -C parameter.
```
for (int i = 0; i < num_threads; i++) {
  pthread_create(&threads[i], NULL, cpu_stress_function, (void*)i);
}
```
- Compute-Intensive Tasks: Each thread performs compute-intensive operations such as floating-point calculations, memory read/write, which consume CPU resources.
```
void* cpu_stress_function(void* arg) {
  while (true) {
      // Perform some CPU-intensive calculations
      double a = 3.14159265358979;
      for (int i = 0; i < 1000000; i++) {
          a = a * a * 3.14159;  // Simulate a calculation
      }
  }
  return NULL;
}
```
- Thread Synchronization and Management: Functions like pthread_join() ensure correct execution and synchronization of threads.
stressapptest DDR Stress Principle: DDR memory stress testing involves extensive memory allocation, access, and data exchange, achieved through high memory usage, frequent memory read/write operations, and memory bandwidth consumption.
- Memory Allocation: In memory stress testing, stressapptest allocates memory of a specified size based on the -M parameter (e.g., -M 8192).
```
void* allocate_memory(size_t size) {
  void* ptr = malloc(size);  // Allocate memory of specified size
  if (ptr == NULL) {
      perror("Memory allocation failed");
      exit(1);
  }
  return ptr;
}
```
- Memory Read/Write Operations: stressapptest performs extensive read and write operations on the allocated memory region. Below is a simple example of memory read/write operations:
```
void stress_memory(void* ptr, size_t size) {
  volatile char* data = (volatile char*)ptr;
      for (size_t i = 0; i < size; i++) {
              data[i] = (char)(i % 256);  // Write data
              char temp = data[i];  // Read data
      }
}
```
- Thread Access to Memory: stressapptest uses multiple threads to increase memory pressure. Multiple threads concurrently access memory, simulating memory usage under high load.
Command Analysis: After running the stress test script, execute the following command: stressapptest -s "$stime" -M "$memory_size" -f /tmp/sat.io1 -f /tmp/sat.io2 -i "$io_threads" -m 8 -C 2 -W
- -s "$stime": This parameter specifies the duration of the test.
- -M "$memory_size": This parameter specifies the amount of memory usage. In the source code, the -M parameter controls the size of memory allocation.
- -f /tmp/sat.io1 -f /tmp/sat.io2: The -f parameter specifies files used for I/O testing.
- -i "$io_threads": This parameter specifies the number of threads used for I/O operations.
- -m 8: This parameter controls the intensity of memory stress.
- -C 2: This parameter specifies the number of CPU cores used for stress testing.
- -W: This parameter enables write operations, meaning both memory reads and writes will be performed.

2. BPU Stress Test

Test Objective: Run the run-portion.sh script and use the tc_hbdk3 tool to adjust the -b (BPU core) and -p (BPU load ratio) parameters to simulate different BPU load scenarios, ensuring the BPU remains stable and achieves expected performance under high load.
Test Purpose: Ensure that the BPU outputs correct results matching expectations when executing computational tasks, and verify that the BPU remains stable during prolonged high-load operation without crashes or errors.
tc_hbdk3 BPU Stress Principle: By loading models containing extensive computational tasks and leveraging BPU for accelerated computing, BPU stress testing is achieved.
Command Analysis: After running the stress test script, execute the following command: tc_hbdk3 -t $1 -b $2 -f $HBM_FILE -i $SRC_FILE -n $MODEL_NAME -o $OUTPUT_0_0,$OUTPUT_0_1,$OUTPUT_1_0,$OUTPUT_1_1 -g 0 -c 0
- -t $1: Passes the value of the $portion parameter, allowing flexible control over the intensity of the stress test and selection of appropriate load levels based on different testing requirements.
- -b $2: Specifies the number of BPU cores to use.
- -f $HBM_FILE: The -f parameter specifies the HBM file.
- -i $SRC_FILE: This file contains input data used for model inference or training; the program reads this data and passes it to the BPU for computation.
- -n $MODEL_NAME: The -n parameter specifies the name of the model to be loaded and executed.
- -o $OUTPUT_0_0: These output files are used to save computation results or log data.

4.5.3.2. Preparations

1. Before starting the stress test, a heatsink must be attached to the chip; otherwise, the chip may enter thermal protection mode, affecting test results.

2. Confirm the integrity of files located in the /app/platform_samples/chip_base_test/01_cpu_bpu_ddr path:

01_cpu_bpu_ddr/
├── models
│   └── HBDK3_MODEL_2K
│       ├── input_0_feature_1x1088x2048x3_ddr_native.bin
│       ├── run.sh
│       └── x5
│           └── gen_I2004_ForTest2k_1x1088x2048x3.hbm
├── scripts
│   ├── run-portion.sh
│   ├── stop_test.sh
│   ├── stress_test.sh
│   └── stressapptest
└── tc_hbdk3

4.5.3.3. Test Method

The stress test script supports the -h suffix to display descriptions of command parameters:

./stress_test.sh -h

Usage: ./stress_test.sh [options]

Options:
  -t <time>        Set the test duration (e.g., 2h for hours, 30m for minutes; default: 48h).
  -m <size>        Set the memory size for stress test in MB (default: 100).
  -i <threads>     Set the I/O threads for stress test (default: 4).
  -b <bpu_core>    Specify the BPU core to use (default: 0).
  -p <portion>     Set the BPU portion value (default: 100).
  -o <directory>   Set the output directory for logs (default: ../../log).
  -h, --help       Show this help message and exit.

Example:
  ./stress_test.sh -t 24h -m 200 -i 8 -p 80

Parameter explanations are as follows:

t <time>: Sets the duration of the stress test. Time format can be hours (e.g., 2h) or minutes (e.g., 30m), with a default value of 48 hours.
m <size>: Sets the memory size for the stress test in MB, defaulting to 100MB.
i <threads>: Sets the number of I/O threads, defaulting to 4.
b <bpu_core>: Specifies the BPU (Processing Unit) core number, defaulting to 0.
p <portion>: Sets the BPU load ratio, defaulting to 100 (i.e., full load).
o <directory>: Sets the directory for log output, defaulting to ../../log.
h, --help: Displays help information and exits the script.

Example: For instance, use the command ./stress_test.sh -t 24h -m 200 -i 8 -p 80 to run a 24-hour stress test using 200MB of memory, 8 I/O threads, and 80% BPU load.

After completing the preparations, run the test command:

cd /app/platform_samples/chip_base_test/01_cpu_bpu_ddr/scripts

./stress_test.sh

The stress test script monitors the operation of stressapptest using the hrut_somstatus command. The log at script startup appears as follows:

Running stress test with the following configuration:
  Stress Time: 48h hours (2880 seconds)
  Memory Size: 100 MB
  I/O Threads: 4
  BPU Core: 0
  BPU Portion: 100%
  Output directory: /app/platform_samples/chip_base_test/log
Starting CPU stress test...
nohup: redirecting stderr to stdout
Starting BPU stress test...
=====================1=====================
temperature-->
        DDR      : 84.0 (C)
83087
        BPU      : 82.9 (C)
        CPU      : 83.7 (C)
cpu frequency-->
              min(M)    cur(M)  max(M)
        cpu0: 300       1500    1500
        cpu1: 300       1500    1500
        cpu2: 300       1500    1500
        cpu3: 300       1500    1500
        cpu4: 300       1500    1500
        cpu5: 300       1500    1500
        cpu6: 300       1500    1500
        cpu7: 300       1500    1500
bpu status information---->
              min(M)    cur(M)  max(M)  ratio
        bpu0: 500       1000    1000    98
ddr frequency information---->
              min(M)    cur(M)  max(M)
        ddr:  266       4266    4266
GPU gc8000 frequency information---->
              min(M)    cur(M)  max(M)
        gc8000:  200    1000    1000

Explanation of hrut_somstatus command results:

temperature: Represents the current temperature of DDR, BPU, and CPU.
cpu frequency: Represents the minimum, current, and maximum operating frequencies (in MHz) for CPUs 0–7.
bpu status information: Represents the minimum, current, and maximum operating frequencies of the BPU, where “ratio” indicates the current BPU utilization.

Run the top command to check CPU utilization:

Each line displays the status of a CPU core in the format:

%CpuX  :  user/system  idle [ progress bar ]

%CpuX: Indicates the CPU core number; for example, %Cpu0 refers to core 0.
user: The proportion of CPU time used for user-space processes (e.g., applications, processes).
system: The proportion of CPU time used for kernel-space processes (kernel operations).
idle: The proportion of time the core is idle; for cpu0, current idle time is 1%.
progress bar: Displays the core’s load status. The length of each bar reflects the intensity of CPU load.

For example, on the first core %Cpu0, 50.8% of the time is spent executing user processes, 48.7% on kernel operations (system processes), and 0% idle time—indicating the core is almost never idle.

4.5.3.4. Test Metrics

After the test program starts, two log files, bpu-stressX.log and cpu-stressX.log, are generated in the /app/platform_samples/chip_base_test/log directory to record the system status during stress testing. The following conditions must be maintained throughout the test:

Stable operation for 48 hours without reboots or system hangs.
Use the following command to check for abnormal prints such as fail, error, or timeout in the log files:

cd "/app/platform_samples/chip_base_test/log/" && grep -iE 'error|fail|timeout' bpu-stress*.log cpu-stress*.log

Run the top command to check CPU and BPU utilization, which should remain stably within the 98–100% range under normal conditions.

Test Results

Enter the command:

cd "/app/platform_samples/chip_base_test/log/" && grep -iE 'error|fail|timeout' bpu-stress*.log cpu-stress*.log

Output shown:

cpu-stress1.log:1970/01/01-02:11:11(UTC) Stats: Completed: 80944.00M in 14.24s 5685.54MB/s, with 0 hardware incidents, 0 errors
cpu-stress1.log:1970/01/01-02:11:11(UTC) Status: PASS - please verify no corrected errors
cpu-stress2.log:1970/01/01-00:10:29(UTC) Stats: Completed: 3300956.00M in 589.29s 5601.62MB/s, with 0 hardware incidents, 0 errors
cpu-stress2.log:1970/01/01-00:10:29(UTC) Status: PASS - please verify no corrected errors
cpu-stress3.log:1970/01/01-00:34:09(UTC) Stats: Completed: 236090.00M in 44.79s 5270.69MB/s, with 0 hardware incidents, 0 errors
cpu-stress3.log:1970/01/01-00:34:09(UTC) Status: PASS - please verify no corrected errors

No other abnormal messages appear in the output, and the number of errors is 0, indicating the stress test results are normal.