Skip to main content

8.4 Multimedia Processing and Applications

This section answers common questions about video encoding/decoding, audio processing, and other multimedia features on D-Robotics RDK boards.

Video Encoding and Decoding​

Q1: The board reports an error when decoding an RTSP video stream (as shown below). What could be the cause?​

Q1: The board reports an error when decoding an... diagram

A: RTSP video stream decoding errors are commonly caused by the following issues and solutions:

  1. The stream is missing PPS and SPS parameter information:
    • Cause: RTSP streams pushed by the streaming server (especially H.264 format) must contain PPS (Picture Parameter Set) and SPS (Sequence Parameter Set) parameter information. The decoder needs this information to parse the video correctly.
    • Solution:
      • If you use ffmpeg to stream from a video file (such as .mp4 or .avi), it is recommended to add the -bsf:v h264_mp4toannexb (H.264 Bitstream Filter: MP4 to Annex B) option to the command (note: in newer versions of ffmpeg, -vbsf has been replaced by -bsf:v). This filter automatically adds PPS and SPS information to the stream. ffmpeg streaming command example: 
        ffmpeg -re -stream_loop -1 -i xxx.mp4 -c:v copy -bsf:v h264_mp4toannexb -f rtsp rtsp://192.168.1.195:8554/h264_stream
        (Replace xxx.mp4 with your video file name, and replace the RTSP server address rtsp://192.168.1.195:8554/h264_stream with the actual address.)
  2. Resolution support limitations:
    • Currently, RDK boards may only support decoding RTSP video streams up to certain resolutions, such as 1080p (1920x1080). Please confirm whether your RTSP stream resolution is within the supported range. Refer to the documentation for your specific board model for the accurate support list.
  3. Streaming software compatibility:
    • VLC direct streaming is not recommended: Using VLC to push RTSP streams directly may fail to be decoded by RDK, because VLC may not support actively adding or ensuring PPS and SPS information during streaming in some configurations. It is recommended to use ffmpeg or other professional streaming tools that ensure complete stream parameters.

Q2: When using ffmpeg for hardware encoding on S100/S600, ghosting occurs at 1080p resolution?​

The root cause lies in the ffmpeg source implementation: when copying input data from AVFrame.data to the v4l2 buffer, because of the hardware IP's 16-byte alignment requirement, the amount of data copied each time is based on the aligned size (for example, the Y plane is copied from 1920*1080 to 1920*1088), and copying the extra bytes ultimately causes encoding ghosting.

For cases where the input file does not meet the hardware IP byte-alignment requirement, ffmpeg provides a video filter option that supports per-frame processing of video frames before encoding, such as scaling, cropping, and alignment. Therefore, you can use the following command to encode and ensure the result matches expectations.

ffmpeg -f rawvideo -pix_fmt yuv420p -s:v 1920x1080 -r 30 -i input.yuv -vf "pad=1920:1088:0:0" -codec:v h264_v4l2m2m -b:v 5M -f h264 output.h264

Common Audio Issues​

Q3: The examples use tinyalsa. What do the parameters mean and how do I use them?​

A: tinyalsa is a lightweight audio library mainly used for Android and embedded Linux systems. It provides a simplified interface to ALSA (Advanced Linux Sound Architecture), making audio development easier for developers. The following are commonly used tinyalsa commands and their parameter meanings:

  1. List all sound cards: 
    tinymix -l
    This command lists all recognized sound cards and their controls in the system. If tinymix -l does not work, you can run cat /proc/asound/cards directly to check.
  2. List controls for a specific sound card: 
    tinymix -c <card_number> -l
    Where <card_number> is the sound card index. This command lists all controls for the specified sound card. Depending on the tinymix version, the required command may differ; you can also try commands such as tinymix -D <card_number> controls.
  3. Get the value of a specific control: 
    tinymix -c <card_number> <control_name>
    This command displays the current value of a control on the specified sound card. For example: 
    tinymix -c 0 'ADC PGA Gain'
    This displays the current value of the control named ADC PGA Gain on sound card 0.
  4. Set the value of a specific control: 
    tinymix -c <card_number> <control_name> <value>
    This command sets the value of a control on the specified sound card. For example: 
    tinymix -c 0 'ADC PGA Gain' 80%
    This sets the control named ADC PGA Gain on sound card 0 to 80%.
  5. View the current audio status: 
    tinymix -c <card_number> -s
    This command displays the current audio status of the specified sound card, including the current values of all controls.
  6. Play an audio file: 
    tinyplay <file_name>
    This command plays the specified audio file. <file_name> is the path and name of the audio file. For example: 
    tinyplay /path/to/audio.wav
    This plays the specified audio file.
  7. Record audio: 
    tinycap <file_name> -D <card_number> -d <device_number> -c <channels> -b <bit_depth> -r <sample_rate> -p <period_size> -n <periods> -t <duration>
    This command records audio and saves it to the specified file. Parameter meanings:
    • <file_name>: Name of the recorded audio file.
    • -D <card_number>: Specify the sound card index.
    • -d <device_number>: Specify the device index (usually the PCM device).
    • -c <channels>: Specify the number of recording channels (for example, 2 for stereo, 4 for four channels).
    • -b <bit_depth>: Specify the audio bit depth (for example, 16 for 16-bit).
    • -r <sample_rate>: Specify the sample rate (for example, 48000 for 48 kHz).
    • -p <period_size>: Specify the period size (in frames).
    • -n <periods>: Specify the number of periods.
    • -t <duration>: Specify the recording duration (in seconds).
    • Example:
    tinycap ./recorded_audio.wav -D 0 -d 1 -c 2 -b 16 -r 48000 -p 512 -n 4 -t 5
    This command uses device 1 on sound card 0 to record 2-channel 16-bit 48 kHz audio for 5 seconds and saves it as recorded_audio.wav.

Q4: How do I distinguish and use USB sound cards and onboard sound cards on RDK boards, especially when multiple audio devices are connected?​

A: When both an onboard sound card (for example, via an audio expansion board) and a USB sound card are connected to an RDK board, the Linux audio system (ALSA) assigns different sound card indices to them. You need to know the correct sound card index to control a specific audio device precisely.

  1. View recognized sound cards and their indices: Use the following command to list all recognized sound cards in the system along with their indices and names:

    cat /proc/asound/cards

    Example output (assuming the USB sound card registers first and the onboard sound card registers second):

     0 [RC08          ]: USB-Audio - ROCWARE RC08
                          ROCWARE RC08 at usb-xhci-hcd.2.auto-1.2, high speed
     1 [duplexaudio   ]: simple-card - duplex-audio
                          duplex-audio

    In this example:

    • USB sound card ROCWARE RC08 is assigned sound card index 0.
    • Onboard sound card duplexaudio (this is usually the name of the RDK audio expansion board) is assigned sound card index 1.
    • Note: Sound card index assignment may change depending on device insertion order, driver loading order, and other factors. If the USB sound card is inserted after system boot, it may receive a larger index.

  2. Use amixer or tinymix to operate on a specific sound card:

    • When you use tools such as amixer (ALSA Mixer command-line utility) or tinymix to view or adjust audio parameters, if you do not specify the card and device numbers, they usually operate on sound card 0 by default.
    • To operate on a specific sound card, use the -c <card_number> (or -c<card_number>) parameter to specify the sound card index, and possibly -D hw:<card_number> or -d <device_number> as needed.
    • View controls for a specific sound card (for example, the onboard sound card with index 1 in the example above): 
      amixer -c 1 controls
      # Or use the hardware device name: amixer -D hw:1 controls
    • Get or set the value of a control on a specific sound card (for example, get the value of the first control named 'ADC PGA Gain' on onboard sound card 1): 
      amixer -c 1 sget 'ADC PGA Gain',0
      To set a value, use sset instead of sget, for example: amixer -c 1 sset 'ADC PGA Gain',0 80%.

Using the methods above, you can accurately identify and control different audio devices connected to RDK boards.

Q5: How to support audio features through the graphical interface?​

  1. Configure the correct sound card device.

    The default fragment_size set when the PulseAudio server starts does not meet the 64-byte alignment required by PDMA, so the default configuration must be modified to ensure the PulseAudio service loads successfully.

    The current board configuration is as follows: /etc/pulse/default.pa

        .ifexists module-udev-detect.so
        load-module module-alsa-sink device=hw:0,1 format=s16le rate=48000 channels=2 channel_map=front-left,front-right mmap=false tsched=0 fragments=2 fragment_size=1920
        load-module module-alsa-source device=hw:0,0 format=s16le rate=48000 channels=2 channel_map=front-left,front-right mmap=false tsched=0 fragments=2 fragment_size=1920
        # load-module module-udev-detect // comment out

    In the configuration above, X in device=hw:X,Y represents the sound card number and Y represents the device number. By default, hw:0,1 is used as the playback device and hw:0,0 as the recording device. If the actual sound card and device numbers match the defaults, no changes are needed; otherwise, update the corresponding device parameter according to the actual device information.

    For how to confirm sound card/device numbers, see Control Commands.

  2. Ensure the audio driver loads before the PulseAudio service.

    PulseAudio can load and manage a sound card only after the corresponding audio driver has been initialized. Therefore, make sure the audio driver finishes loading before the PulseAudio service starts.

    The audio driver can be loaded dynamically or built-in.

    For dynamic loading, you can modify the file: /usr/bin/hobot-loadko.sh

    For driver loading commands, see Sound Card Debugging.

  3. After modification, save the configuration and restart the system. Once the audio driver is loaded, the graphical interface features will work normally.

  1. Configure the correct sound card device.

    The default fragment_size set when the PulseAudio server starts does not meet the 64-byte alignment required by PDMA, so the default configuration must be modified to ensure the PulseAudio service loads successfully.

    The current board configuration is as follows: /etc/pipewire/pipewire-pulse.conf.d/99-s600-audio.conf 

    pulse.cmd = [
        {
            cmd = "load-module"
            args = "module-alsa-sink device=hw:0,1 format=s16le rate=48000 channels=2 channel_map=front-left,front-right mmap=false tsched=0 fragments=2 sink_properties='priority.session=1200'"
            flags = [ ]
        }
        {
            cmd = "load-module"
            args = "module-alsa-source device=hw:0,0 format=s16le rate=48000 channels=2 channel_map=front-left,front-right mmap=false tsched=0 fragments=2 source_properties='priority.session=2500'"
            flags = [ ]
        }
    ]

    In the configuration above, X in device=hw:X,Y represents the sound card number and Y represents the device number. By default, hw:0,1 is used as the playback device and hw:0,0 as the recording device. If the actual sound card and device numbers match the defaults, no changes are needed; otherwise, update the corresponding device parameter according to the actual device information.

    For how to confirm sound card/device numbers, see Control Commands.

    Note: If the playback and recording device numbers are the same, for example /dev/snd/pcmC0D1p and /dev/snd/pcmC0D1c, you need to specify sink_name=XXX and source_name=XXX in the args respectively to clearly distinguish the input and output devices.

    pulse.cmd = [
        {
            ...
            args = "module-alsa-sink device=hw:0,0 sink_name=alsa_output.hw_0_0 ...
            ...
        }
        {
            ...
            args = "module-alsa-source device=hw:0,0 source_name=alsa_input.hw_0_0 ...
            ...
        }
    ]

  2. Ensure the audio driver loads before the PulseAudio service.

    PulseAudio can load and manage a sound card only after the corresponding audio driver has been initialized. Therefore, make sure the audio driver finishes loading before the PulseAudio service starts.

    The audio driver can be loaded dynamically or built-in.

    For dynamic loading, you can modify the file: /usr/bin/hobot-loadko.sh

    For driver loading commands, see Sound Card Debugging.

  3. After modification, save the configuration and restart the system. Once the audio driver is loaded, the graphical interface features will work normally.

Last updated on