WonderMedia?
API Developer
Guide
Revision 0.3
November 27, 2009
WonderMedia? Technologies Inc.
WonderMedia? Technologies Inc.
November 27, 2009 i
Revision History
Revision Date Revision Descriptions RD
0.1 2009/10/20 Initial version Willy Chuang
0.2 2009/10/26
- Add “1.3.4 Color Space Convert”
- Add “2.6 Motion JPEG”
- Correct some typos
Willy Chuang
0.3 2009/11/27
- Add “2.3.3 About VDU Scaling”
- Add “2.3.4. Real Time and Recursive Scaling”
- Add “2.5.5 PD (Partial Decoding)”
Willy Chuang
WonderMedia? Technologies Inc.
November 27, 2009 ii
Table of Contents
REVISION HISTORY..........................................................................................................................................................I
TABLE OF CONTENTS...................................................................................................................................................... II
LIST OF FIGURES.............................................................................................................................................................III
LIST OF TABLES................................................................................................................................................................III
1. INTRODUCTION............................................................................................................................................................ 1
1.1. PURPOSE.................................................................................................................................................................... 1
1.1.1. Abbreviation................................................................................................................................................ 1
1.2. WONDERMEDIA API CONCEPT.................................................................................................................................. 1
1.3. HARDWARE BLOCK OVERVIEW ON WONDERMEDIA SOC........................................................................................... 2
1.3.1. Video Path - Real Time Scaling ............................................................................................................ 2
1.3.2. Video Path – Recursive Scaling ............................................................................................................ 3
1.3.3. Graphic Path ............................................................................................................................................... 3
1.3.4. Color Space Convert (CSC).................................................................................................................... 4
1.4. WONDERMEDIA API RELEASE PACKAGE.................................................................................................................... 4
1.4.1. Build Application with WMT API............................................................................................................ 5
1.5. OPEN SOURCE LICENSES............................................................................................................................................ 6
1.6. WMTAPI VERSION................................................................................................................................................... 7
2. VIDEO DECODER - JPEG (WMT JDEC) .................................................................................................................... 8
2.1. OVERVIEW................................................................................................................................................................. 8
2.1.1. Class Diagram............................................................................................................................................ 8
2.1.2. Frame Buffer Structure........................................................................................................................... 9
2.1.3. Basic Concepts......................................................................................................................................... 10
2.1.4. Basic Flow and Memory Usage ........................................................................................................... 12
2.2. WMT JDECAPI INITIALIZE.................................................................................................................................... 14
2.3. DECODE JPEG......................................................................................................................................................... 15
2.3.1. APIs for Settings before Decoding .................................................................................................... 15
2.3.2. APIs for Decoding ................................................................................................................................... 19
2.3.3. About VDU Scaling ................................................................................................................................. 21
2.3.4. Real Time and Recursive Scaling....................................................................................................... 24
2.4. DISPLAY JPEG......................................................................................................................................................... 26
2.4.1. Display Mode ............................................................................................................................................ 26
2.4.2. WMT View Structure ............................................................................................................................... 28
2.5. ADVANCE JDEC FEATURES ..................................................................................................................................... 30
2.5.1. Allocate Physically Continuous Memory .......................................................................................... 30
2.5.2. MSD (Multi-Segment Decoding)......................................................................................................... 31
2.5.3. More Detail about JDEC......................................................................................................................... 33
2.5.4. JPEG Rotation........................................................................................................................................... 35
2.5.5. PD (Partial Decoding)............................................................................................................................ 36
2.6. MOTION JPEG (MJPG)............................................................................................................................................ 39
WonderMedia? Technologies Inc.
November 27, 2009 ii i
List of Figures
FIGURE 1 –WMTAPIARCHITECTURE ......................................................................................................................................... 2
FIGURE 2 –WMTHARDWARE BLOCK ARCHITECTURE.................................................................................................................. 3
FIGURE 3 –WMTAPI FILE TREE ................................................................................................................................................. 5
FIGURE 4 –WONDERMEDIA API (HEADER FILES) HIERARCHY...................................................................................................... 5
FIGURE 5 – CLASS DIAGRAM OF WONDERMEDIA API RELATED WITH JDEC................................................................................. 8
FIGURE 6 – FRAME BUFFER ARCHITECTURE ................................................................................................................................ 12
FIGURE 7 –WONDERMEDIA BSPMEMORY MAP......................................................................................................................... 13
FIGURE 8 – DATA PATH OF HARDWARE JPEG DECODER ............................................................................................................... 14
FIGURE 9 – FLOW OF SETTING SCALING ...................................................................................................................................... 18
FIGURE 10 – DECODE TO ORIGINAL DIMENSION ........................................................................................................................... 21
FIGURE 11 – DECODE TO 1/4 DIMENSION OF ORIGINAL DIMENSION .............................................................................................. 22
FIGURE 12 – VDU SCALING ALGORITHM .................................................................................................................................... 22
FIGURE 13 – IDEA OF SCALING ALGORITHM ................................................................................................................................ 23
FIGURE 14 – EXAMPLE OF WMT_JDEC_FILE_IN_SIZE() ................................................................................................................ 24
FIGURE 15 – FLOW OF REAL TIME SCALING................................................................................................................................. 24
FIGURE 16 – FLOW OF RECURSIVE SCALING + REAL TIME SCALING.............................................................................................. 25
FIGURE 17 – DISPLAY MODE: IF PICTURE IS SMALLER THAN DISPLAY AREA.................................................................................. 27
FIGURE 18 – DISPLAY MODE: IF PICTURE IS LARGER THAN DISPLAY AREA.................................................................................... 28
FIGURE 19 – VDO_VIEW_T INTRODUCTION ................................................................................................................................. 29
FIGURE 20 – FIRST SCENARIO OF PD........................................................................................................................................... 37
FIGURE 21 – SECOND SCENARIO OF PD....................................................................................................................................... 37
FIGURE 22 – SOFTWARE ARCHITECTURE FOR JPEG AND MJPEG ................................................................................................ 39
FIGURE 23 – SOFTWARE ARCHITECTURE OF MPLAYER FOR MJPEG............................................................................................ 40
List of Tables
TABLE 1 – OPEN SOURCE USED IN LINUX BSP .............................................................................................................................. 6
TABLE 2 – RELATIONSHIP BETWEEN SOURCES WITH DECODED COLORS ....................................................................................... 17
WonderMedia? Technologies Inc.
November 27, 2009 1
1. Introduction
1.1. Purpose
This document provides a concept of idea for customers to develop application on Linux operation system based on
WonderMedia? SoC platform. For this purpose, customer should understand WonderMedia? APIs framework and interface in
detail. The interface includes video decoder, e.g., JPEG, video post processing, graphic acceleration, security engine (cipher)
(option) and so on. The contents of this document are suitable for all WonderMedia? SoC named WM8xxx series in theory.
However, they are not suitable for old VT8xxx SoC.
1.1.1. Abbreviation
BSP Board Support Package
CSC Color Space Convert
EVB EValuation Board
FB Frame Buffer, a physically continuous memory buffer used to store video data
GE Graphic Engine
GOV Graphic OVerlay
GOVR Graphic OVerlay (Read function)
GOVW Graphic OVerlay (Write function)
IDE Integrated Development Environment
JDEC Jpeg DECoder
HW Hardware
MB Memory Block, a Linux driver for application to access physically continuous memory buffer
MSD Multi-Segment Decoding
MVL MontaVista? Linux®
PD Partial Decoding
SDK Software Development Toolkit
SE Security Engine
SW Software
VDMA Video DMA, a hardware block in WonderMedia? Soc
VDU Video Decode Unit, a hardware block in WonderMedia? SoC
VPP Video Post Processing function block
VPU Video Post Unit, a hardware block in WonderMedia? Soc
WMT WonderMedia? Technologies
1.2. WonderMedia? API Concept
WonderMedia? API, “WMT API”, is a set of functions provided by WonderMedia? for application developers, who
would develop their product on WonderMedia? platform. Basically WMT APIs access almost hardware blocks, which are
related with multimedia, though Linux drivers. It is very important for customers to be proficient in WMT API if they must
develop applications on WonderMedia? platform. Roughly we may divide WMT APIs into some groups. They are “SE”,
“VDU”, “GE”, “MB”, “VDMA” and “VPP”.
“VPP” is an abbreviation of “Video Post Processing”. It has not a real hardware block. In generally, VPP we called
contains “VPU”, “GOV (R/W)” and “SCALER” hardware blocks.
“VPU” is a hardware block used for further scaling up/down images in frame buffers and send them “GOV (R/W)”
blocks.
WonderMedia? Technologies Inc.
November 27, 2009 2
“VDU” means “Video Decode Unit”. It is used to decode video bitstreams, (e.g., JPEG, MPEG, H.264 and so on),
which depends on SoC capabilities. The output data of VDU should be saved on a physically continuous memory buffer,
named “video frame buffer”. We also call this “video frame buffer” generated by VDU as “video plane”.
“GOV” is an important hardware block for graphic overlay and alpha mixing. In some WonderMedia? Soc, there are
two blocks called “GOVW” and “GOVR” for “write” and “read”, respectively. “GOVW” mixes graphic plane and video
planes with alpha blending and write the result to DRAM. Then “GOVR” reads the data on DRAM and send it to output
device.
“GE” means “Graphic Engine”. It is used to handle graphic acceleration by hardware. We call the plane operated by
GE as “graphic plane”.
“Cipher” or “SE” (Security Engine) is a hardware block, which customer may use to encrypt or decrypt data.
“VDMA” is a hardware block used to copy or “bitblt” video data.
“MB” means “Memory Block”, which is a software block (Linux driver) used to allocate physically continuous
memory buffer for user-space applications.
WonderMedia? API
Linux Application
JPEG/
MJPEG
Security
Engine
Graphic
VPP Engine
User Space
Kernel Space
JPEG
Driver
Cipher
Driver
GE
Driver
VPP
Driver
Figure 1 – WMT API Architecture
1.3. Hardware Block Overview on WonderMedia? SoC
In this subsection, we will introduce WonderMedia? Soc hardware block architecture. One thing reader must keep in
mind is that this figure below is roughly overview and there may be a little difference between different WonderMedia? Soc.
1.3.1. Video Path - Real Time Scaling
WonderMedia? Technologies Inc.
November 27, 2009 3
Figure 2 below shows a simple user scenario about video and graphic data path. Application must allocate “Video
FB#1” and inform “VDU” and “VDU” will render decoded video data on it. Then application “sends” “Video FB#1” to
“VPU”. In property timing, “GOVW” will overlay “Graphic FB” and “Video FB#1” and copy them to “Video FB#3” or
“Video FB#4” automatically. Sometimes, dimension of “Video FB#1” is difference with “Video FB#3” because of scaling
by “VPU”. In this scenario, hardware scaling is finished by “VPU” and “GOVW” automatically. We call it as “Real time
scaling”.
1.3.2. Video Path – Recursive Scaling
In Figure 2 below, if application does not send “Video FB#1” to “VPU” directly. Instead, it sends “Video FB#1”
(source buffer) and “Video FB#2” (destination buffer) to “SCALER”. Then “SCALER” scales “Video FB#1” onto “Video
FB#2”. (LOOK OUT: These two frame buffers were allocated by application in advance.) After that, application “sends”
“Video FB#2” to “VPU” and “VPU” and “GOVW” “copy” “Video FB#2” to “Video FB#3” without scaling by hardware
automatically. In this scenario, hardware scaling is finished by “SCLAER”. We call it as “Recursive scaling”.
GE
VPU
SCLAER GOVW GOVR Output
Device
System
Memory
Input VDU
Device
Video FB #1 Video FB #2 Grapihc FB
Video FB #4
Video FB #3
Recursive
Scaling
Real Time
Scaling
Figure 2 – WMT Hardware Block Architecture
In software’s point of view, the “Video FB#1” and “Video FB#2” must be allocated in user space, including
applications or WMT API. The “Video FB#3” or “Video FB#4” are allocated by Linux driver in kernel space. As for
“Graphic FB”, it is pre-allocated before system was booted. Moreover, all frame buffers above should be physically
continuous memory since they are accessed by hardware block.
It is important to know that the data formats of video FB supported by WonderMedia? hardware design are
“YUV4:2:0”, “YUV4:2:2H”, “YUV4:4:4” and “ARGB” only. In other word, if user copy or move data to frame buffer by
other format, the result will be correct because hardware cannot recognizes it.
1.3.3. Graphic Path
In software engineer’s point of view, application should be access “Graphic FB” directly. In general user scenario,
“Graphic FB” is always access by popular graphic libraries, e.g., DirectFB®. And application calls APIs of popular graphic
libraries.
Current graphic libraries which WonderMedia? adopt in Linux BSP is “GTK+/DirectFB®” and “VGUI”, which is
proprietary library developed by WonderMedia?. (You may get more information about DirectFB® or GTK+ from hyperlink
below)
DirectFB stands for Direct Frame Buffer. It is a software library for the GNU/Linux operating system that provides
"hardware graphics acceleration, input device handling and abstraction, integrated windowing system with support for
translucent windows and multiple display layers, not only on top of the Linux Framebuffer Device." It is very popular for
Linux operating system and open source, we adopt it as our graphic library in SDK to handle graphic effect. Besides, there
WonderMedia? Technologies Inc.
November 27, 2009 4
are many tutorial documents or articles about DirectFB and user can study or get any information, which he/she wants,
easily. (For example, refer to http://www.directfb.org/docs/DirectFB_Reference/ ).
GTK+ is a highly usable, feature rich toolkit for creating graphical user interfaces which boasts cross platform
compatibility and an easy to use API. (Please refer to http://www.gtk.org/ )
1.3.4. Color Space Convert (CSC)
There are two hardware blocks have CSC capabilities inside, they are “VDU” and “GOVR”. “VDU” has only one
CSC function (from YUV to RGB in JFIF algorithm). But the “GOVR” have two CSC functions (from YUV to RGB or
RGB to YUV in some algorithms).
In theory, the CSC algorithm in “VDU” and “GOVR” in hardware design should be the same. If developers want to
verify video quality by dumping video data directly, they may let “VDU” to decode into ARGB mode (“Video FB #1”) and
let “VPU” bypass to “GOVR” buffer (“Video FB#3”) (Note: bypass means no scaling in “VPU”). In this case, “Video FB
#1” and “Video FB #3” are the same. Then developers may dump “Video FB#1” by “ut_jdec” or “Video FB #3” by
“ut_vpp”, respectively. As for how to dump it by unit test programs, please refer to another document.
1.4. WonderMedia? API Release Package
Formal WMT API in WonderMedia? Linux BSP release package includes following components.
|-- wmt_api\
|-- release\
|-- include\
|-- com-jdec.h (file)
|-- com-mb.h (file)
|-- common-api.h (file)
|-- com-se.h (file)
|-- com-vd.h (file)
|-- com-vdma.h (file)
|-- com-video.h (file)
|-- wmt-api.h (file)
|-- wmt-jdec.h (file)
|-- wmt-mb.h (file)
|-- wmt-se.h (file)
|-- wmt-vd.h (file)
|-- wmt-vdma.h (file)
|-- wmt-vpp.h (file)
|-- lib\
|-- libwmtapi.a (file)
|-- libwmtapi.so (file)
|-- libwmtapi.so.1.0.1 (file)
WonderMedia? Technologies Inc.
November 27, 2009 5
Figure 3 – WMT API File Tree
The WonderMedia? API package may shorten as
|-- release
|-- include\
|-- com-xxx.h (files)
|-- wmt-xxx.h (files)
|-- lib\
|-- libwmtapi.a (file)
|-- libwmtapi.so (file)
Here, libwmtapi.a is used for static link and libwmtapi.so is used in shared library. The files with prefix “com-” mean
they are used by both user space (API) and kernel space (Linux drivers). And the files with prefix “wmt-“ means these
header files are used by applications in user space only.
Following figure describes the relationship between “com-xxx.h” with “wmt-xxx.h”.
com-video.h
(vdo_ )
com-vd.h
(vd_ )
com-jdec.h
(jdec_ )
com-mpeg.h
com-vpp.h
(vpp_ )
Only define following structures
vdo_framebuf_t
vdo_view_t
Basic "class" file for all
video decoders
wmt-jdec.h
(wmt_jdec_ )
Files above this line are used for
both kernel and user spaces
Files below this line are used for
user space only
nd_api.h
(nd_ )
wmt-vpp.h
(wmt_vpp_ )
com-ge.h
(ge_ )
wmt-se.h
(wmt_se_ )
wmt-ge.h
(wmt_ge_ )
com-se.h
(se_ )
WMT API
#include(): Usage: (a-page-name-you-want-to-include[,title|,notitle])
Figure 4 –WonderMedia? API (Header Files) Hierarchy
1.4.1. Build Application with WMT API
To build applications, which are related with WMT API, user must have a correct setting of “wmt_api”. The related
paths of developer’s environment may be different for everybody. For example, assume your application is “ND SDK”,
which folder is “nd_sdk_2.2.0_090618”. To build it successfully, user must confirm whether the paths in Makefile are
correct or not.
API_PATH = ../../wmt_api/release/
WonderMedia? Technologies Inc.
November 27, 2009 6
ARM_PATH = /opt/arm-linux
The environment of WonderMedia? API for this example should be set as below.
|-- nd_sdk\
|-- nd_sdk_2.2.0_090618\
|-- Makefile (file)
|-- wmt_api\
|-- release\
1.5. Open Source Licenses
Reader may get more information about these libraries from hyperlink below. All source codes in WMT API are
proprietary and developed by WonderMedia?. These codes provide an interface for application to access WonderMedia?
hardware blocks through Linux driver. Moreover, the APIs is independent with application or Linux drivers. It is an
independent library. Therefore, it is not open source.
However, there are many kinds of multimedia SoC provided by WonderMedia?. The following table shows open source
licenses, which libraries may be used in WonderMedia? Linux SDK for demonstration purpose. Customers need to
understand these open source licenses if they want to adopt them to develop their application and products.
Module License
GTK+ LGPL
DirectFB LGPL
MPlayer GPL
FFMPEG LGPL
Linux kernel (MVL 5.0) GPL + MVL
Table 1 – Open Source Used in Linux BSP
Reader may get more information about these libraries from hyperlink below.
MPlayer is a free and open source media player distributed under the GNU General Public License. The program is
available for all major operating systems, including Linux and other OS. MPlayer supports a wide variety of media formats.
In addition to its wide range of supported formats MPlayer can also save all streamed content to a file. (Please refer to
MPlayer website http://www.mplayerhq.hu/ )
FFmpeg is a complete solution to record, convert and stream audio and video. It includes libavcodec, the leading
audio/video codec library. FFmpeg is developed under Linux, but it can be compiled under most operating systems,
including Windows. (Please refer to http://ffmpeg.mplayerhq.hu/ )
DirectFB stands for Direct Frame Buffer. It is a software library for the GNU/Linux operating system that provides
"hardware graphics acceleration, input device handling and abstraction, integrated windowing system with support for
translucent windows and multiple display layers, not only on top of the Linux Framebuffer Device." It is very popular for
Linux operating system and open source, we adopt it as our graphic library in SDK to handle graphic effect. Besides, there
are many tutorial documents or articles about DirectFB and user can study or get any information, which he/she wants,
easily. (For example, refer to http://www.directfb.org/docs/DirectFB_Reference/ ).
GTK+ is a highly usable, feature rich toolkit for creating graphical user interfaces which boasts cross platform
compatibility and an easy to use API. (Please refer to http://www.gtk.org/ )
WonderMedia? Technologies Inc.
November 27, 2009 7
1.6. WMT API Version
Current latest version of WMT API is “4.0.3”. If user’s application is limitation some specified version of WMT API,
it is better to call the version-check API below to avoid some trouble.
For example, if your application requires WMT API “4.0.0” or later version, it is better to add sample codes below in
your application.
if( wmt_api_ver_require("4.0.0") ){
return -1;
}
To understand detail release notes, user may refer to “wmt-api.h”, which is under “/release/include/” of WMT API.
WonderMedia? Technologies Inc.
November 27, 2009 8
2. Video Decoder - JPEG (WMT JDEC)
In this chapter, we will mainly introduce how to use WMT API (JPEG decoding related).
2.1. Overview
2.1.1. Class Diagram
Figure 5 – Class Diagram of WonderMedia? API related with JDEC
In applications developer’s point of view, the most important “class” in figure above is “jdec_frameinfo_t”. Almost
application would use or operate this structure. Sometimes, “jdec_hdr_info_” may be used by some applications. As for
other structures, they are always used inside WMT JDEC API. Application developer may ignore them at first stage.
WonderMedia? Technologies Inc.
November 27, 2009 9
2.1.2. Frame Buffer Structure
The structure “vdo_framebuf_t” is the core structure in all video related API. Developer must know this structure in
detail.
Structure: vdo_framebuf_t
Syntax
typedef struct {
unsigned int y_addr;
unsigned int c_addr;
unsigned int y_size;
unsigned int c_size;
unsigned int img_w;
unsigned int img_h;
unsigned int fb_w;
unsigned int fb_h;
unsigned int bpp;
vdo_color_fmt col_fmt;
unsigned int h_crop;
unsigned int v_crop;
} vdo_framebuf_t;
Description
This structure describes all settings of one frame buffer
y_addr: physical address of this frame buffer for Y (or RGB) plane
y_size: buffer size of this frame buffer for Y (or RGB) plane
c_addr: physical address of this frame buffer for C plane
c_size: buffer size of this frame buffer for C plane
img_w: valid picture width on this frame buffer for both Y (or RGB) /C planes
img_h: valid picture height on this frame buffer for both Y (or RGB) /C planes
fb_w: the scan line offset of this frame buffer in width for both Y(or RGB) /C planes
fb_h: the lines of this frame buffer in height for both Y(or RGB) /C planes
bpp: bits per pixel on Y (or RGB)/C plane
col_fmt: color format on Y (or RGB)/C plane
h_crop: cropping on X coordinate
v_crop: cropping on Y coordinate
Note
If the color format of this frame buffer is RGB, c_addr and c_size would be both zero.
And y_addr and y_size will indicate the address and buffer size of RGB plane,
respectively. Following figure shows the concepts of some members in this structure.
y_addr or c_addy points to left-top.
WonderMedia? Technologies Inc.
November 27, 2009 1 0
In almost case, h_crop and v_crop are usually zero.
This structure is defined in WMT API (com-video.h).
For application, this structure “vdo_framebuf_t” is not enough since there is no user space address. Application
cannot do anything with this structure. So we create a new structure called “jdec_frameinfo_t”, which is a “subclass” of
“vdo_framebuf_t” and used in almost WMT JDEC API.
Structure: jdec_frameinfo_t
Syntax
typedef struct {
vdo_framebuf_t fb;
unsigned int y_addr_user;
unsigned int c_addr_user;
unsigned int scaled;
} jdec_frameinfo_t;
Description
This structure describes all settings of one frame buffer used for JPEG driver only. This
structure should be thought as a “subclass” of vdo_framebuf_t class.
fb: For more detail, please refer to definition in vdo_framebuf_t.
y_addr_user: logical address of this frame buffer for Y (or RGB) plane in user space
c_addr_user: logical address of this frame buffer for C plane in user space
scaled: means the image on this frame buffer whether was scaled by JPEG decode
driver. (0: original size; 1: scaled) (Please do NOT modify this value in application)
Note This structure is defined in and shared with WMT API (com-jdec.h) instead of ND API.
2.1.3.Basic Concepts
To be proficient in WMT JPEG API, some basic concepts about decoding JPEG are necessary for customers.
Profile
According to JPEG specification in “ISO/IEC 10918-1”, there are many kinds of profiles. However, WonderMedia?
only supports “Baseline DCT” profile hardware decoding. (Symbol: SOF0 and Marker: 0xFF 0xC0) If customers try to use
WMT API to decode non-baseline JPEG file, they may get following error message.
WonderMedia? Technologies Inc.
November 27, 2009 1 1
- The profile of JPEG does not be supported
SOLUTION: Please use software JPEG decoder to decode this file
Resolution
They are some kinds of definitions about “resolution”. To avoid confusion, we define them as
Source resolution: It means the resolution of source pictures. For example, the source resolution of one JPEG file
with dimension 600x400 is 600x400.
Decoded resolution: It means resolution of the picture decoded by hardware. In theory, it should be equal to source
resolution, but they may be different if customers request decoder to scale it down. WonderMedia? JPEG
hardware decoder may decode a JPEG file to its original, 1/2, 1/4 or 1/8 size.
Frame buffer resolution: It means resolution of a memory buffer, which stores decoded JPEG. Because of hardware
alignment limitation, it should be closed to but larger than decoded resolution.
Display resolution: It means output resolution of screen. In theory, it should not be any arbitrary value. It must fit
VESA specification, e.g., “XGA (1024x768)” or “SVGA (800x600)”.
There no certain relationships between these four resolutions. It depends on customer’s applications.
Color Space/Color Format
The term “color space or color format” means horizontal and vertical sampling in a JPEG file, e.g., YUV 4:4:4 or
4:2:2H. In general speaking, WonderMedia? support all color space in JPEG source file. They include YUV 4:2:0, 4:1:1,
4:2:2H, 4:2:2V; 4:4:4 and gray level. However, the decoded color space of WonderMedia? hardware decoder only supports
YUV4:2:0, YUV4:2:2H, YUV4:4:4 or RGB 32 bit modes. Source color space and decoded color space may be different.
Customers may use some WMT API to do that if necessary.
Scaling
They are some kinds of definitions about “scaling”. To avoid confusion, we define them as
Hardware decoding scaling: It means the scaling function of WonderMedia? JPEG hardware decoder. (Only 1/1, 1/2,
1/4 and 1/8 are supported).
Hardware scaler scaling: It means the scaling functions of WonderMedia? SCL444 hardware block. It is called by
“recursive scaling”.
VPP scaler scaling: It means the scaling functions of WonderMedia? VPU hardware block. It is called by “real time
scaling”.
About “recursive scaling” and “real time scaling”, please refer to “subsection 1.3”.
Frame Buffer
The term “Frame Buffer (FB)” here means a memory buffer with continuous physical address and it is used to store
decoded JPEG data or other video data.
Basically, frame buffer is almost used by hardware components. Therefore there is alignment limitation in line size (or
line width). However, it should be transparent for application developer.
In WMT JDEC API, there are two main color spaces, i.e., YUV and RGB. For YUV, WMT JDEC API allocates two
frame buffers for one picture. They are called “Y-Plane”, which saves Y data (luminance), and “C-Plane”, which stores Cb
WonderMedia? Technologies Inc.
November 27, 2009 1 2
and Cr data (chrominance). The bytes per line of Y-plane and C-plane are the same with source width of picture if it
satisfies hardware limitation (64 bytes alignment). As for the bytes per line of RGB plane, it is four times of source width of
picture (64 bytes alignment). For example, if source picture is 1000x480, then decoded frame buffer should be like one of
two cases below.
Y-Plane
Y0 Y1 Y2 Y3
C-Plane
Cb Cr RGB-Plane
B G R A
1024 bytes 4032 bytes
1000 bytes 4000 bytes
Figure 6 – Frame buffer architecture
One thing which developer should know is the format of ARGB. It is 32 bits and offset 0 is blue (B), offset 1 is green
(G), offset 2 is red (R) and offset 3 is alpha value (A), which is not used for JPEG. (The order is similar with Microsoft
BMP format in 32 bits mode). When developers should use YUV or RGB mode, it depends on their application. Please
keep in mind that application has better performance, lower memory bandwidth and good quality in YUV mode if the data
source is originally in YUV mode, e.g., JPEG file.
2.1.4. Basic Flow and Memory Usage
Generally speaking, there are standard steps to finish a JPEG decoding job. They are respectively
Allocate a buffer to save source data called “source data buffer”, which may be NOT physically continuous
Do correct settings, e.g., decoded color, scaling…
Allocate a frame buffer to save destination data called “destination video frame buffer”, which should be
physically continuous
Start hardware JPEG decoding to (1/1, 1/2, 1/4 or 1/8)
Scaling by “Recursive Scaling Mode” to desired resolution (Option)
Rotate video frame buffer (Option)
Display this picture if this decoding is successful
The order of these steps is the same with application order to decode JPEG.
Allocate a buffer to save source data:
Suppose the resolution of source image file, which user wants to decode, is 4096x4096 in YUV4:4:4 mode. The size
without compression is about 4096x4096x3 = 50,331,648 = 48 MB. Assume the compression ratio is 50%, we need 24 MB
to store this compressed file. So the size of “source data buffer” is 24 MB. The pseudo code in your application is
filesize = 25165824; /* 24 MB */
fp = fopen(“test.jpg”, "rb"); /* Open image file */
WonderMedia? Technologies Inc.
November 27, 2009 1 3
src_buf = malloc(filesize); /* Allocate source data buffer */
fread(src_bu, 1, filesize, fp); /* Copy data to src_buf */
However, 24 MB is not easy to be allocated in embedded system. To guarantee the decoding process is successful if
system memory is shortage, we provide a method called “MSD (Multi-Segment Decoding). We will introduce MSD in
detail later.
Do correct settings:
Important settings about JPEG decoding are “decoded color” and “scaling ratio”. They will impact the memory size of
“destination video frame buffer”. We will introduce it in subsection below later.
Allocate a frame buffer to save destination data:
Since frame buffer to save destination data, i.e., “destination video frame buffer”, should be physically continuous,
developers cannot guarantee they may always allocate it each time. In WMT API architecture, application must allocate
physically continuous memory by MB (Memory Block) driver. The video frame buffer is configured in Linux Menuconfig.
(Default is 32 MB)
Because DRAM on platform is limited, developers cannot decode too large JPEG file, though the upper limitation of
WonderMedia? hardware JPEG is 65536x65536.
Pre-allocated
Memory
Memory Pool
For Linux
kernel &
application
Allocated
Memory by
MB
Figure 7 – WonderMedia? BSP Memory Map
We could divide total memory (DRAM) into three parts, “Pre-allocated Memory”, “Allocated memory by MB” and
“Memory Pool for Linux kernel and application”. “Pre-allocated Memory” is mainly used for frame buffer device driver,
i.e., graphic layer. “Allocated memory by MB” is mainly used for both video decoders and VPP. If there is no any display
device on product, these two blocks above are not necessary. The rest memory is used for Linux kernel and applications.
Develops must fully understand how many memory their application need on their product.
Follow the example above. If user wants to decode this file as original size in ARGB mode, then the frame buffer size
needed is 4096x4096x4 = 67,108,864 = 64 MB. This decoding will be fail if there is no 64 MB physically continuous
memory. However, in most embedded products in current markets, the resolution of output device is smaller than full HD
(1920x1080). So we may let JPEG hardware decoder to scale this picture down to 2048x2048 automatically by doing
correct settings. Then the video frame buffer size required will be reduced to 2048x2048x4 = 16,777,216 = 16 MB. 16 MB
is more possible to be allocated successfully in embedded produce than 64 MB. The key concept is the maximum decoded
resolution is dependent on MB buffer size configured in Linux kernel. The more MB buffer size you reserve, the bigger
decoded resolution is. Understand this concept is very important for developers. After all, embedded product is not personal
computer. The cost of DRAM is also a considered factor.
WonderMedia? Technologies Inc.
November 27, 2009 1 4
Figure 8 – Data Path of hardware JPEG decoder
Start hardware JPEG decoding:
There are some WMT JDEC APIs for application to decode JPEG. We will introduce it in subsection below later.
Scaling by “Recursive Scaling Mode” to desired resolution:
From Figure 7 above, application must allocate another video frame buffer, e.g., “Destination Video FB #2”, as
destination buffer of “SCALER”. In most case, the size of “Video FB #2” is smaller than “Video FB #1”. (For example, the
resolution of “Video FB #1” may be 2048x2048, but “Video FB #2” is only “1024x768”.) After scaling finished, “Video FB
#1” may be released immediately. This step is option. Not all application needs this step.
Rotate video frame buffer:
This step is option. Not all application needs this step. We will introduce it in subsection below later.
Display this picture:
There are some WMT JDEC APIs for application to display a JPEG frame buffer. We will introduce it in subsection
below later.
2.2. WMT JDEC API Initialize
In this subsection, we will discuss about how to initialize drivers if we want to use hardware JPEG decoder. In WMT
API architecture, to use hardware JPEG decoder, user must initialize VPP (video post process) and VD (common video
decoders) drivers. Following is easiest sample codes to initialize/release drivers for JPEG decoders.
#include "wmt-vpp.h"
#include "wmt-vd.h"
#include "wmt-jdec.h"
int wmt_init(vd_handle_t *handle)
{
wmt_vpp_init();
if( wmt_vd_init(handle, VD_JPEG) != 0) {
printf("*E* Video Decoder Initial FAIL!\n");
return -1;
}
wmt_vpp_govw_set_ge_visible(0); /* Turn off GE layer */
WonderMedia? Technologies Inc.
November 27, 2009 1 5
wmt_vpp_govw_set_vpu_visible(1); /* Turn on VPP layer */
return 0;
} /* End of wmt_init() */
int wmt_exit(vd_handle_t *handle)
{
wmt_vpp_govw_set_ge_visible(1); /* Turn on GE layer */
wmt_vpp_govw_set_vpu_visible(0); /* Turn off VPP layer */
wmt_vd_exit(handle);
wmt_vpp_exit();
return 0;
} /* End of wmt_exit() */
int main(void)
{
vd_handle_t jpeg_handle;
wmt_init(&jpeg_handle);
wmt_exit(&jpeg_handle);
return 0;
}
In WMT API architecture, JPEG decoded frame buffer is put into video path, i.e., this frame buffer will be as input
buffer of VPU instead of GE. So, to see image in video path, we must turn on VPU and turn off GE. And we must turn off
VPU and turn on GE to return to default status. In almost application, default case is enable graphic layer and disable video
layer. LOOK OUT: If user turns on both graphic and video layer, he/she must take care of overlay settings carefully. We
will introduce this topic about overlay in another chapter.
The behavior of “vd_handle_t” is similar with standard I/O API. User must keep this handle until it is released
because it will be used in each WMT JDEC APIs. For multi-thread applications, each thread should initialize and release
drivers and own itself handle.
In the sample codes above, reader may find that there are three header files are included, “wmt-vpp.h”, “wmt-vd.h”
and “wmt-jdec.h”
“wmt-vpp.h” is necessary for WMP VPP API if developers want to display frame buffer.
“wmt-vd.h” is common video decoders. It is necessary for JPEG decoder since JPEG decoder is a “subclass” of
command VD (video decoder).
“wmt_jdec.h” is necessary for application to use WMT JPEG API.
2.3. Decode JPEG
We had discussed the seven steps about JPEG decoding in previous subsection briefly. In this subsection, we will
introduce the steps 2 ~ 4 in detail.
2.3.1. APIs for Settings before Decoding
When user wants to decode a JPEG file, he/she must do correct settings. If he/she will not do any settings, API inside
will use default value. However, default value may be not what user wants in some cases. So we recommend that
application developer always call settings APIs to set correct ones.
WonderMedia? Technologies Inc.
November 27, 2009 1 6
Decoded Color
In theory, user may set decoded color as one of enumeration “vdo_color_fmt”. However, there are two kinds of
decoded color which are recommended, they are “VDO_COL_FMT_AUTO” and “VDO_COL_FMT_ARGB”. If
application does not allocate the “destination video frame buffer”, e.g., allocated by API automatically, it may not care it too
much. Otherwise, application must care about bpp (bits per pixel) is different with each color format. For instance, bpp is 4
for ARGB; 3 for YUV4:4:4; 2 for YUV4:2:2H and 1.5 for YUV4:2:0.
Enum: vdo_color_fmt
Syntax
typedef enum {
VDO_COL_FMT_YUV420,
VDO_COL_FMT_YUV422H,
VDO_COL_FMT_YUV422V,
VDO_COL_FMT_YUV444,
VDO_COL_FMT_YUV411,
VDO_COL_FMT_GRAY,
VDO_COL_FMT_ARGB,
VDO_COL_FMT_AUTO,
VDO_COL_FMT_MAX,
VDO_COL_FMT_UNKNOWN
} vdo_color_fmt;
Description
This enum is used and shared for all WonderMedia? APIs to set color format on a frame
buffer.
YUV color format (VDO_COL_FMT_YUV420, VDO_COL_FMT_YUV422H,
VDO_COL_FMT_YUV422V, VDO_COL_FMT_YUV444, VDO_COL_FMT_YUV411):
For all YUV color format, we will use 2 planes layout, i.e., Y and C planes.
Gray level color format (VDO_COL_FMT_GRAY): Although it is gray level, but
we still allocate a C plane with full 0x80 data on it because of our hardware limitation.
RGB color format (VDO_COL_FMT_ARGB): For RGB space, we only support RGB
32 bits, i.e., ARGB where A is a dummy byte.
VDO_COL_FMT_AUTO: The value is used for JPEG only.
For WonderMedia? SoC, JPEG hardware decoder may decode a JPEG file to YUV 420,
422H, 444 or ARGB color format for different source format. The best way to set decoded
color format is set to VDO_COL_FMT_AUTO or VDO_COL_FMT_ARGB. In auto mode,
JPEG decoder driver will decode YUV 420, 411 source to 420, YUV 444 source to 444.
Other source color formats will be decoded to YUV 422H. In RGB mode, JPEG decoder
driver will decode it to RGB 32 bits no matter what kind of source format.
Note This enumeration is defined in WonderMedia? API (com-video.h).
Application may call the API below to set decoded color.
int wmt_jdec_set_dc_format(
vd_handle_t *handle,
vdo_color_fmt decoded_color)
Currently WonderMedia? hardware decoder only supports 3 components (YUV) in JPEG file (RGB or CMYK
components are not supported). Therefore, we assume the source format of JPEG file is always in YUV domain. So if user
set decoded color as “VDO_COL_FMT_AUTO”, the JPEG driver will decode JPEG to YUV. If user wants data in RGB
domain, please set “VDO_COL_FMT_ARGB”.
Source color format Set Decoded Color Result
YUV 4:2:0/4:1:1 VDO_COL_FMT_AUTO VDO_COL_FMT_YUV420
WonderMedia? Technologies Inc.
November 27, 2009 1 7
VDO_COL_FMT_ARGB VDO_COL_FMT_ARGB
YUV 4:2:2H/4:2:2V VDO_COL_FMT_AUTO VDO_COL_FMT_YUV422H VDO_COL_FMT_ARGB VDO_COL_FMT_ARGB
YUV 4:4:4 VDO_COL_FMT_AUTO VDO_COL_FMT_YUV444 VDO_COL_FMT_ARGB VDO_COL_FMT_ARGB
Others Not Supported Not Supported
Table 2 – Relationship between sources with decoded colors
JPEG Decoder Scaling
As we talk above, WonderMedia? JPEG hardware decoder support scaling down feature (original, 1/2, 1/4 and 1/8). To
save memory, let JPEG hardware decoder to decode and scale down together is helpful.
Enum: vd_scale_ratio
Syntax
typedef enum {
SCALE_ORIGINAL,
SCALE_HALF,
SCALE_QUARTER,
SCALE_EIGHTH,
SCALE_BEST_FIT,
SCALE_THUMBS,
SCALE_MAX
} vd_scale_ratio;
Description
This enum is used and shared for all WMT JDEC APIs to set scaling ratio for hardware
decoder.
SCALE_ORIGINAL: Do not scale source and keep its original size
SCALE_HALF: Let hardware decoder scale down JPEG to 1/2 size in both width and
height.
SCALE_QUARTER: Let hardware decoder scale down JPEG to 1/4 size in both width and
height.
SCALE_EIGHTH: Let hardware decoder scale down JPEG to 1/8 size in both width and
height.
SCALE_BEST_FIT: Let WMT JDEC API decide the scaling ratio according to source
size and resolution of display device.
SCALE_THUMBS: It is a customize mode. User may input any value as width and height.
Note This enumeration is defined in WonderMedia? API (com-jdec.h).
Application may call the API below to set scaling ratio.
int wmt_jdec_set_scale(vd_handle_t *handle, vd_scale_ratio ratio)
Although there are six choices, which user may set, in this API, only four values of these six are set to hardware
registers. User must keep one thing in mind that the decoded frame buffer size is only 1/1, 1/2, 1/4 or 1/8 even he/she
chooses “Best Fit” or “Thumbs”.
WonderMedia? Technologies Inc.
November 27, 2009 1 8
Application WMT JDEC
API
JDEC
Driver
HW JPEG
Decoder
SCALE_ORIGINAL/
SCALE_HALF/
SCALE_QUARTER/
SCALE_EIGHTH/
SCALE_BEST_FIT/
SCALE_THUMBS
SCALE_ORIGINAL/
SCALE_HALF/
SCALE_QUARTER/
SCALE_EIGHTH/
Figure 9 – Flow of setting scaling
For example, assume source JPEG size is 4000x4000 and resolution of display device is “1024x768”. If user set
scaling mode as “Best Fit”, hardware JPEG decoder may decodes it to 1000x1000 or 2000x2000 according to scaling
algorithm, which user set or default value in WMT JDEC API.
Enum: jpeg_scale_alg
Syntax
typedef enum {
JPEG_SA_SMART,
JPEG_SA_QUALITY,
JPEG_SA_FAST,
JPEG_SA_MAX
} jpeg_scale_alg; /* SA = Scale Algorithm */
Description
This enum is used and shared for all WMT JDEC APIs to set scaling algorithm.
JPEG_SA_SMART: Use JPEG_SA_QUALITY or JPEG_SA_FAST automatically
according to picture size and requested size
JPEG_SA_QUALITY: Decode to a larger size than requested size by HW decoder. Then
use HW SCALER to scaling down to requested size (better quality).
JPEG_SA_FAST: Decode to a smaller size than requested size by HW decoder. Then use
HW SCALER to scaling up to requested size (better performance).
Note This enumeration is defined in WonderMedia? API (wmt-jdec.h).
The default value of scaling algorithm in WMT JDEC API is “JPEG_SA_SMART”. For example, in the example
above, WMT API request hardware to decode source to 1000x1000 because the smart algorithm decides scaling ratio by
“Distance”.
Distance[ (1000x1000) – (1024x768) ] < Distance[(2000x2000) – (1024x768)]
If users do not want to be affected by this uncertain factor, they must set scaling ratio as 1/1, 1/2, 1/4 or 1/8 correctly
in wmt_jdec_set_scale( ).
Get JPEG Information
WonderMedia? Technologies Inc.
November 27, 2009 1 9
Sometimes it is useful to know header information of JPEG before starting to decode JPEG. For example, if
application does not know picture resolution and resolution of display device, it cannot know how to set scaling ratio.
Following WMT JDEC API is used for application to parse JPEG header.
int wmt_jdec_header (
unsigned char *buf_in,
unsigned int buf_size,
jdec_hdr_info_t *hdr);
Structure: jdec_hdr_info_t
Syntax
typedef enum {
JT_FRAME,
JT_FIELD_TOP,
JT_FIELD_BOTTOM
} jpeg_type;
typedef struct {
jpeg_type type;
int field_size;
} avi1_t;
typedef struct {
unsigned int profile;
unsigned int sof_w;
unsigned int sof_h;
unsigned int filesize;
vdo_color_fmt src_color;
avi1_t avi1
} jdec_hdr_info_t;
Description
profile: It means the JPEG file coding profile. Only Baseline DCT (0xFF 0xC0) is
supported by WonderMedia? JPEG decoder
sof_w: The value means valid image width in JPEG header.
sof_h: The value means valid image height in JPEG header.
filesize: The value means the file size (in bytes) of JPEG file.
src_color: It means the color format in JPEG header.
avil: The value means “avi1” in APP. It used for MJPEG only. In some MJPEG, one
JPEG frame is divided into two fields. We need this information when decode this kind of
frame.
Note This structure is defined in WonderMedia? API (com-jdec.h).
2.3.2. APIs for Decoding
There are four basic and easy WMT JDEC APIs for user to decode one JPEG file.
int wmt_jdec_buf(
vd_handle_t *handle,
unsigned char *buf_in,
unsigned int bufsize,
jdec_frameinfo_t *frameinfo);
int wmt_jdec_file(
vd_handle_t *handle,
WonderMedia? Technologies Inc.
November 27, 2009 2 0
char *filename,
jdec_frameinfo_t *frameinfo);
int wmt_jdec_buf_in_size(
vd_handle_t *handle,
unsigned char *buf_in,
unsigned int bufsize,
unsigned int request_w,
unsigned int request_h,
jpeg_scale_alg scale_alg,
jdec_frameinfo_t *frameinfo);
int wmt_jdec_file_in_size(
vd_handle_t *handle,
char *filename,
unsigned int request_w,
unsigned int request_h,
jpeg_scale_alg scale_alg,
jdec_frameinfo_t *frameinfo);
The common point among these four APIs is the last parameter “jdec_frameinfo_t *frameinfo”, which
means a physically continuous frame buffer. Users must know that this frame buffer was allocated automatically inside
these APIs, so it must be release by users by calling API wmt_jdec_fb_free() when this frame buffer is useless.
int wmt_jdec_fb_free (vd_handle_t *handle, jdec_frameinfo_t *info);
All information in this frame buffer are set by driver and returned to user. In general, user should not modify it.
Following shows the simplest sample code to decode JPEG file.
int main(void)
{
vd_handle_t handle;
jdec_frameinfo_t jfb;
/* Initialization */
wmt_init(&handle);
/* Start decoding if you have file path */
wmt_jdec_file(&handle, filename, &jfb);
/* Or, start decoding if you have a buffer */
/* Display the JPEG */
wmt_jdec_show_image(&handle, &jfb, JPEG_DM_BEST_FIT);
/* Free JPEG frame buffer */
wmt_jdec_fb_free(&handle, &jfb);
/* Release resource */
wmt_exit(&handle);
return 0;
}
About wmt_jdec_show_image (), we will introduce it in next subsection.
WonderMedia? Technologies Inc.
November 27, 2009 2 1
In this sample code, it uses all default settings for JPEG decoder. In this case, user does not know what scaling ratio
and decoded color are. This picture will be best fit to screen. (The scaling ratio is automatically generated by WMT API
according to source dimension and resolution of display device.)
You may replace “wmt_jdec_file()” and “wmt_jdec_buf()” in the simplest sample code above by
“wmt_jdec_file_in_size()” and “wmt_jdec_buf_in_size()”. The only difference between these two scenarios
is in scaling ratio. Scaling ratio of second case is generated according to your input request_w and request_h. Inside
these two WMT API in second case, the performance is worse since it does twice frame buffer “copy”, one is by “VDU”,
another is “recursive scaling” by “SCALER”. Although the performance is worse, these two APIs are useful for user to
create “Thumbnail”. For example, if you want to decode any JPEG to a 256x192 thumbnail, you just need to call
wmt_jdec_file_in_size(&handle, filename, 256, 192,
JPEG_SA_SMART, &jfb);
or
wmt_jdec_buf_in_size(&handle, buf, bufsize, 256, 192,
JPEG_SA_SMART, &jfb);
In theory, these values “request_w” and “request_h” could be arbitrary number. However, 16-alignment is
recommended and these two values must be larger than 64 because of hardware limitation. If they are not 16-alignment, the
final result may be a little difference with request values. For example, if these two values are 157 and 133, the result
generated by hardware may be 144 and 128.
2.3.3. About VDU Scaling
Recall the scaling algorithm (“JPEG_SA_SMART, JPEG_SA_QUALITY, or JPEG_SA_FAST”) mentioned in
previous subsection. This scaling algorithm only impacts VDU scaling ratio. If application does not call
“wmt_jdec_set_scale( )” or call it with “SCALE_BEST_FIT” as input argument before calling
“wmt_jdec_file( )”, then the default scaling algorithm in “wmt_jdec_file( )” is “”JPEG_SA_SMART”.
int wmt_jdec_set_scale(vd_handle_t *handle, vd_scale_ratio ratio)
If application calls “wmt_jdec_set_scale( )” with input argument “SCALE_ORIGINAL, SCALE_HALF,
SCALE_QUARTER or SCALE_EIGHTH,” before calling “wmt_jdec_file( )”, then VDU will decode and scale it
with this setting. It is not difficult to understand it.
Figure 10 – Decode to original dimension
WonderMedia? Technologies Inc.
November 27, 2009 2 2
Figure 11 – Decode to 1/4 dimension of original dimension
Now let us focus on best fit mode case. For best fit mode, how JPEG driver decide exact scaling ratio for hardware for
“JPEG_SA_SMART” scaling algorithm? We introduce it in previous subsection. The key point is “Distance” between
source dimensions with display resolution. For example, assume there is a JPEG file with dimension 4000x4000.
2000x2000
1000x1000
500x500
2000x2000
1000x1000
500x500
1024x768
640x480
Figure 12 – VDU scaling algorithm
If resolution of current display area is 1024x768, the nearest distance to 1024x768 is 1000x1000. Then VDU will
decode and scaling to 1/4 automatically in best fit mode. If resolution of current display area is 640x480, the nearest
distance to 640x480 is 500x500. Then VDU will decode and scaling to 1/8 automatically in best fit mode. This example
shows how “JPEG_SA_SMART” algorithm works inside WMT API “wmt_jdec_file( )”.
The behavior of “wmt_jdec_buf( )” is the same with “wmt_jdec_file( )” besides some different input
arguments in these two APIs.
But how “wmt_jdec_file_in_size( )” works? In fact, there are 3 major steps in this function.
wmt_jdec_file_in_size(
vd_handle_t *handle,
char *filename,
unsigned int request_w,
unsigned int request_h,
jpeg_scale_alg scale_alg,
jdec_frameinfo_t *frameinfo)
= wmt_jdec_set_scale ( ) + wmt_jdec_file( ) + wmt_jdec_stretch( )
An exact scaling ratio will be calculated according to “scale_alg” and fill it in “wmt_jdec_set_scale( )”
“wmt_jdec_file( ) requests VDU to do scaling with this value (1/1, 1/2, 1/4 or 1/8)
WonderMedia? Technologies Inc.
November 27, 2009 2 3
“wmt_jdec_stretch( )” scales the decoded frame buffer to requested dimension (request_w x request_h)
(it was called by recursive scaling mode).
We discussed the case above if “scale_alg” is equal to “JPEG_SA_SMART”. Essentially, “JPEG_SA_
SMART” is equal to “JPEG_SA_FAST” or “JPEG_SA_QUALITY” according to “Distance”, which is the same concept
above. For example, assume there is a JPEG file with dimension 4000x4000 and requested in
“wmt_jdec_file_in_size( )” is 1280x1024.
4000x4000
2000x2000
1000x1000
500x500
Requested
dimension
VDU Scaling to
(JPEG_SA_QUALITY)
VDU Scaling to
(JPEG_SA_FAST)
Figure 13 – Idea of scaling algorithm
[Scenario 1] “scale_alg” = “JPEG_SA_FAST”: VDU uses 1/4 scaling ratio, which is smaller and nearest to
requested dimension.
[Scenario 2] “scale_alg” = “JPEG_SA_QUALITY”: VDU uses 1/2 scaling ratio, which is larger and nearest to
requested dimension.
[Scenario 3] “scale_alg” = “JPEG_SA_SMART”: VDU uses 1/4 scaling ratio (same as “JPEG_SA_FAST” in
this case) because the “Distance” of scenario 1 is nearer to requested dimension than scenario 2.
SCLAER
System
Memory
VDU
CPU
Decoded
Frame Buffer
Scaled
Frame Buffer
Source Data
Buffer
Input
Device
4000x4000 2000x2000 768x768
wmt_jdec_file_in_size(handle, filename, 1024, 768,
JPEG_SA_QUALITY, fb);
WonderMedia? Technologies Inc.
November 27, 2009 2 4
Figure 14 – Example of wmt_jdec_file_in_size()
From this figure above, the last variable “fb” in “wmt_jdec_file_in_size( )” points to “Scaled Frame Buffer”.
Why its dimension is 768x768 instead of 1024x768? Because the shape of original JPEG file is “square”. To keep its shape
without torsion, we must decode it to 1024x1024 or 768x768. However, 1024x1024 is out of size than user’s requirement.
Therefore, we decode it to 768x768. As for what users see, it depends on the settings in related display APIs. We will
introduce it in detail in next subsection. LOOK OUT: the “Decoded Frame buffer” was allocated and released inside
“wmt_jdec_file_in_size( )”. User does not know its existence. But the rest memory in MB pool must be larger than
2000x2000 + 768x768 in this example. Otherwise this API will return error because of memory shortage.
From WMT API 4.1.1 or later version, we create a new algorithm called “JPEG_SA_HIGH_QUALITY”. This value
will forces VDU decodes JPEG to original size without any scaling down. However, there is a risk of memory shortage. In
this example, to decode this JPEG file to original size (4000x4000) in ARGB mode, the requested memory is 64 MB. It is
huge. Therefore, if memory shortage, “JPEG_SA_HIGH_QUALITY” will be downgrade to “JPEG_SA_QUALITY”
automatically inside “wmt_jdec_file_in_size( )”.
The behavior of “wmt_jdec_buf_in_size( )” is the same with “wmt_jdec_file_in_size( )” besides some
different input arguments in these two APIs.
2.3.4. Real Time and Recursive Scaling
We introduced “Real time scaling” and “recursive scaling”. We will discuss it in detail in this subsection. Following
figure shows an example of real time scaling.
Figure 15 – Flow of real time scaling
In this example above, we find that the memory bandwidth is consumed by “VPU”, “GOVW” and “GORV”,
respectively. Assume the video format on all frame buffers are ARGB (RGB 32 bits).
VPU: VPU consumes at least 457.76 MBytes per second roughly [ (2000x2000x4 (read from decoded frame buffer)
- N (scale 2000x2000 to 1024x768 runtime)) x 30 ] (Please keep in mind that hardware needs more memory
bandwidth in scaling down than scaling up)
GOVW: GOVW read two buffers (both video and graphic) and write one buffer with 1024x768 30 times in one
second. Then GOVW consumes 247.5 MBytes per second roughly [ (1024x768x4 (read from graphic plane) +
768x768x4 (read from video plane) +1024x768x4 (write))x30 ] (Here we suppose VPU scales 2000x2000 to
768x768)
WonderMedia? Technologies Inc.
November 27, 2009 2 5
GOVR: GOVR read one buffer with 1024x768 60 times in one second. Then GOVR consumes 90 MBytes per
second roughly [ (1024x768x4 (read from display frame buffer) x60 ]
So the roughly total memory bandwidth consumption is 457.76 + 247.5 + 90 + 30xN > 795.26 MB/sec.
Now let us use recursive scaling mode in the same example. Following figure shows this case.
Figure 16 – Flow of recursive scaling + real time scaling
Again, memory bandwidth is consumed by “VPU”, “GOVW” and “GORV”. They are respectively. Assume the video
frame on frame buffer is ARGB (RGB 32 bits)
VPU: VPU consumes 67.5 MBytes per second roughly [ 768x768x4 (read from scaled frame buffer) x 30 ]
GOVW: GOVW read two buffers (both video and graphic) and write one buffer with 1024x768 30 times in one
second. Then GOVW consumes 247.5 MBytes per second roughly [ (1024x768x4 (read from graphic plane) +
768x768x4 (read from video plane) +1024x768x4 (write))x30 ] (Here we suppose VPU scales 2000x2000 to
768x768)
GOVR: GOVR read one buffer with 1024x768 60 times in one second. Then GOVR consumes 90 MBytes per
second roughly [ (1024x768x4 (read from display frame buffer) x60 ]
So the roughly total memory bandwidth consumption is 67.5 + 247.5 + 90 = 405 MB/sec. In this example, we find the
total memory bandwidth of using recursive scaling mode is about 51% (=405/795.26) of no recursive scaling case. (let
N=0).
Although memory bandwidth is reduced enormously by using recursive scaling mode, the side-effect is that
performance is reduced because we must create a new scaled frame buffer. It is a trade-off and depends on customer’s
choice.
The key point of recursive scaling in WMT JDEC API is “wmt_jdec_stretch( )”, which was called by
“wmt_jdec_file_in_size( )” inside.
int wmt_jdec_stretch(
vd_handle_t *handle,
unsigned int request_w,
unsigned int request_h,
jpeg_scale_alg scale_alg,
jdec_frameinfo_t *jfb_in);
The input arguments of this API is same with ones of “wmt_jdec_file_in_size( )”.
WonderMedia? Technologies Inc.
November 27, 2009 2 6
Since recursive scaling is called “recursive”, it means user may scale it many times as they want. For example, assume
someone has a decoded frame buffer with 2000x2000 dimension and he/she wants to scale it down to 300x200. Sample
code below shows how to do it.
Scenario 1: One time recursive>
wmt_jdec_stretch(handle, 300, 200, JPEG_SA_QUALITY, &fb);
Scenario 2: Multi-recursive>
wmt_jdec_stretch(handle, 1000, 1000, JPEG_SA_QUALITY, &fb);
wmt_jdec_stretch(handle, 500, 400, JPEG_SA_QUALITY, &fb);
wmt_jdec_stretch(handle, 300, 200, JPEG_SA_QUALITY, &fb);
In these two scenarios, although the last input argument in each API call is same, i.e., “fb”, its essential was modified
inside “wmt_jdec_stretch( )” and the original old memory buffer was released in this API, too. Therefore, if you want
to keep the original frame buffer, please allocate another frame buffer and copy the data of original frame buffer to new
frame buffer.
2.4. Display JPEG
There are two APIs to display a decoded JPEG file, which data are stored in a “jdec_frameinfo_t” structure.
int wmt_jdec_show_image(
vd_handle_t *handle,
jdec_frameinfo_t *frameinfo,
jpeg_disp_mode mode);
int wmt_jdec_show(
vd_handle_t *handle,
jdec_frameinfo_t *frameinfo,
vdo_view_t *view);
Basically hardware JPEG has no capability of display. They just call WMT VPP API inside these two APIs. Of course,
user may call WMT VPP API directly instead of WMT JDEC API. However, it is better to call WMT JDEC API if your
application is JPEG related.
Next, we will introduce these two APIs in detail.
2.4.1. Display Mode
Enum: jpeg_disp_mode
Syntax
typedef enum {
JPEG_DM_BEST_FIT,
JPEG_DM_Original,
JPEG_DM_FULL_SCREEN,
JPEG_DM_FULL_SCREEN_TORSION,
JPEG_DM_MAX
} jpeg_disp_mode; /* DM = Display Mode */
Description
JPEG_DM_BEST_FIT: If picture is smaller than display area, it will be located on center
of screen without scaling up. Otherwise, it will be scaled down to fit display area without
torsion (remain aspect ratio).
JPEG_DM_Original: Picture will be put at top-left of screen (0,0). If the picture is
WonderMedia? Technologies Inc.
November 27, 2009 2 7
larger than display area, some part will be invisible.
JPEG_DM_FULL_SCREEN: If picture is smaller than display area, it will be scaled up
without torsion (remain aspect ratio). If it is larger than display area, it will be scaled
down to fit display area without torsion (remain aspect ratio).
JPEG_DM_FULL_SCREEN_TORSION: If picture is smaller than display area, it will be
scaled up with torsion (not remain aspect ratio). If it is larger than display area, it will be
scaled down to fit display area with torsion (not remain aspect ratio)..
Note This enumeration is defined in WMT API (wmt-jdec.h).
Following figures shows visual result about these four kinds of display mode in two cases.
If picture is smaller than display area:
In this case, visual result of these four display mode are different each other.
Figure 17 – Display mode: if picture is smaller than display area
If picture is larger than display area:
In this case, visual result of “Best Fit” and “Full Screen” are the same. In “Original” mode, some part, which is out of
display area, is cut to keep its original size.
Display area
Source size
JPEG_DM_BEST_FIT
Display area
JPEG_DM_FULL_SCREEN
Display area
JPEG_DM_Original
Display area
JPEG_DM_FULL_SCREEN_TORSION
WonderMedia? Technologies Inc.
November 27, 2009 2 8
Figure 18 – Display mode: if picture is larger than display area
2.4.2. WMT View Structure
These four display modes mentioned above are satisfied for almost applications. However, these are not enough for
some applications. To support all kinds of possibilities, we define a structure named “vdo_view_t”.
Structure: vdo_view_t
Syntax
typedef struct {
unsigned int resx_src;
unsigned int resy_src;
unsigned int resx_virtual;
unsigned int resy_virtual;
unsigned int resx_visual;
unsigned int resy_visual;
unsigned int posx;
unsigned int posy;
unsigned int offsetx;
unsigned int offsety;
} vdo_view_t;;
This structure describes all settings of one frame buffer used for JPEG driver only. This
structure should be thought as a “subclass” of vdo_framebuf_t class.
resx_src: this value is similar with fb_w in vdo_framebuf_t. It means line size.
resy_src: this value is similar with fb_h in vdo_framebuf_t.
resx_virtual: this value is source width which user wants to be shown
Display area
Source size
JPEG_DM_BEST_FIT
Display area
JPEG_DM_FULL_SCREEN
Display area
JPEG_DM_Original
Display area
JPEG_DM_FULL_SCREEN_TORSION
WonderMedia? Technologies Inc.
November 27, 2009 2 9
resy_virtual: this value is source height which user wants to be shown
resx_visual: this value is destination width on display device
resy_visual: this value is destination height on display device
offsetx: means the start x-coordinate of source image which user wants to be shown
offsety: means the start y-coordinate of source image which user wants to be shown
posx: means the start x-coordinate of destination on display device
posy: means the start y-coordinate of destination on display device
Note This structure is defined in and shared with WMT API (com-video.h).
Developers may refer to figure below to understand in detail about this structure.
Figure 19 – vdo_view_t introduction
In almost applications, the values of (offsetx, offsety) are always (0, 0). Moreover, the values of (resx_virtual,
resy_virtual) and (resx_src, resy_src) are always the same with (img_w, img_h) and (fb_w, fb_h) in “vdo_framebuf_t” or
“jdec_frameinfo_t”.
As for (posx, posy) and (resx_visual, resy_visual), they depends on relationship between source dimension with
resolution of display device. For example, assume the resolution of display device is 1024x768, source image is 400x300.
To scale up full image two times and show it at center on display device, you must set
/* Source */
resx_src = 448; /* 64 alignment of 400 */
resy_src = 304; /* 16 alignment of 300 */
offsetx = 0;
offsety = 0;
resx_virtual = 400;
resy_virtual = 300;
source
resy_src Display Area
resx_src
resx_visual
resy_visual
posx
posy
offsetx
offsety
resx_virtual
resy_virtual
WonderMedia? Technologies Inc.
November 27, 2009 3 0
/* Destination */
resx_visual = 800; /* 400x2 */
resy_visual = 600; /* 300x2 */
posx = 112; /* (1024-800)/2 */
posy = 84; /* (768-600)/2 */
2.5. Advance JDEC Features
2.5.1. Allocate Physically Continuous Memory
In WonderMedia? API architecture, all physically continuous memory related with video frame buffer must be handled
by MB (Memory Block) driver. For application, it may call WMT MB API to allocate/free this kind of memory. However,
we do not recommend user to do that because there are some detail must be deal with, i.e., alignment issue. Therefore, we
provide following APIs in WMT JDEC API for applications.
int wmt_jdec_fb_alloc (
vd_handle_t *handle,
unsigned int pic_w,
unsigned int pic_h,
vdo_color_fmt src_color,
vdo_color_fmt decoded_color_in,
jdec_input_t *arg_in);
int wmt_jdec_fb_free (vd_handle_t *handle, jdec_frameinfo_t *info);
int wmt_jdec_alloc (
vd_handle_t *handle,
unsigned int buf_w,
unsigned int buf_h,
vdo_color_fmt color_fmt,
jdec_frameinfo_t *jfb);
int wmt_jdec_free (vd_handle_t *handle, jdec_frameinfo_t *jfb);
Basically these two free APIs are identical essentially since all parameters in them are the same. The major difference
between two allocate APIs is in returned value, “jdec_input_t” and “jdec_frameinfo_t”, respectively.
The structure “jdec_input_t” is used inside some WMT JDEC API. Therefore, developer do not need to
understand it too mush. They just need to know when wmt_jdec_fb_alloc( ) returned, following members in
“jdec_input_t” would be filled automatically.
dst_y_addr_user: User space address of Y or RGB plane
dst_c_addr_user: User space address of C or RGB plane
dst_y_addr: Physical address of Y or RGB plane
dst_c_addr: Physical address of C plane (Null for RGB)
dst_y_size: Buffer size for Y or RGB plane
dst_c_size: Buffer size for C plane (0 for RGB)
The structure “jdec_frameinfo_t” is very popular among WMT JDEC API. In application developer’s point of
view, calling “wmt_jdec_alloc ()” to allocate physically continuous memory is recommended if they need to do that.
(LOOK OUT: “wmt_jdec_alloc ()” is supported in WMT API version “4.0.3” or later version)
LOOK OUT: Once application gets a physically continuous frame buffer, only user-space address could be accessed
by application, i.e., “dst_y_addr_user/dst_c_addr_user” in “jdec_input_t” or “y_addr_user/c_addr_user” in
“jdec_frameinfo_t” since all application under Linux running in user space. Besides, this memory comes from
WonderMedia? Technologies Inc.
November 27, 2009 3 1
“Allocated Memory by MB” and it has no relationship with “Memory Pool For Linux kernel & application” in Figure 7. In
other word, user cannot find memory shortage if he/she type “free” or “cat /proc/meminfo” Linux command even user had
allocated the video frame buffer. To watch MB memory status, please type “cat /proc/mbinfo”. More detail about MB,
please refer to another chapter “Memory Block (WMT MB)”.
2.5.2. MSD (Multi-Segment Decoding)
We discussed memory usage in previous subsection. (Please refer to “Figure 8 Data Path of hardware JPEG decoder”)
So we know to decode a JPEG file, we need at least two memory buffers, “Source Data Buffer” and “Destination Video
FB”, respectively. Basically, the “Source Data Buffer” size is equal to file size of a compressed JPEG file. Although it was
compressed, its size is still very huge if its quality is good and resolution is high. If we still request “Source Data Buffer”
size must be equal to file size, sometimes JPEG decoding is fail if rest system memory is not enough. To solve this problem,
we provide a method named MSD.
So-called MSD is that we just allocate a small buffer named “Segment Buffer” to store part of compress JPEG file.
For example, if a compress JPEG file is 8.5 MB and segment buffer is 1 MB. Then we must do 9 loops to finish the JPEG
decoding work. In each loop JPEG decoder only decodes 1 MB except the last loop (only 0.5 MB). MSD provides a good
way to decode JPEG when system memory is not enough. However, the performance becomes a little worse if MSD is
enabled. As for how large is better for segment buffer? Since the larger the size of segment buffer is, the better the
performance is. We recommend it is 1 MB or larger.
To enable/disable MSD, please use following API.
Syntax
int wmt_jdec_msd_enable(
vd_handle_t *handle,
unsigned int fileszie,
unsigned int segment_size,
unsigned int msd_enable)
Parameters
handle [IN]: means the handle of JPEG decoder
fileszie [IN]: means the file size of JPEG file. Or set “0” to ignore it.
segment_size [IN]: means size of segment buffer, 1 MB or larger is recommended.
msd_enable [IN]: means the MSD flag (enable:1 or disable: 0).
Description The API is used to enable/disable MSD for WMT JDEC API.
Return value Return 0 if success; otherwise return an error code (negative value)
Note
Two JDEC APIs, “wmt_jdec_file()” and “wmt_jdec_file_in_size()”, are affected by this API. Sample codes
below shows how to enable/disable MSD.
int main(void)
{
vd_handle_t handle;
jdec_frameinfo_t jfb;
/* Initialization */
wmt_init(&handle);
/* Enable MSD */
wmt_jdec_msd_enable(&handle, 0, 1024*1024, 1);
/* Start decoding if you have file path */
WonderMedia? Technologies Inc.
November 27, 2009 3 2
wmt_jdec_file(&handle, filename, &jfb);
/* Or, start decoding if you have a buffer */
/* Display the JPEG */
wmt_jdec_show_image(&handle, &jfb, JPEG_DM_BEST_FIT);
/* Free JPEG frame buffer */
wmt_jdec_fb_free(&handle, &jfb);
/* Release resource */
wmt_exit(&handle);
return 0;
}
Besides “wmt_jdec_msd_enable ()”, there is another API, “wmt_jdec_segment_write()”, for MSD if user does not
want use “wmt_jdec_file()” and “wmt_jdec_file_in_size()” to decode JPEG.
Syntax
int wmt_jdec_segment_write(
vd_handle_t *handle,
unsigned char *buf_in,
unsigned int bufsize,
unsigned int filesize,
unsigned int seg_no)
Parameters
handle [IN]: means the handle of JPEG decoder
buf_in [IN]: means the address pointed to segment buffer
bufsize [IN]: means size of segment buffer, 1 MB or larger is recommended
filesize [IN]: means the file size of JPEG file
seg_no [IN]: means the segment number (start from 1, not 0)
Description The API is used for MSD only
Return value Return 0 if success; otherwise return an error code (negative value)
Note
Following sample codes shows this new way for MSD.
int main(void)
{
vd_handle_t handle;
jdec_frameinfo_t jfb;
/* Initialization */
wmt_init(&handle);
/* Enable MSD */
seg_size = 1024 * 1024;
wmt_jdec_msd_enable(&handle, filesize, seg_size, 1);
/* Start decoding if you have file path */
segment = 1;
remain_byte = filesize;
WonderMedia? Technologies Inc.
November 27, 2009 3 3
while(remain_byte) {
if( remain_byte <= seg_size) {
read_size = remain_byte;
remain_byte = 0;
}
else {
read_size = seg_size;
remain_byte -= seg_size;
}
if( read_size != fread(buf, 1, read_size, fp) ){
/* Do error handling here */
}
ret = wmt_jdec_segment_write(handle, buf, read_size, filesize,
segment);
if( ret != 0 ){
/* Do error handling here */
}
segment++;
}
/* Wait for decdoing finish after all data are feed */
ret = wmt_jdec_finish(&handle, &jfb);
/* Display the JPEG */
wmt_jdec_show_image(&handle, &jfb, JPEG_DM_BEST_FIT);
/* Free JPEG frame buffer */
wmt_jdec_fb_free(&handle, &jfb);
/* Release resource */
wmt_exit(&handle);
return 0;
}
LOOK OUT: If developers use this way for MSD, the second parameter, “filesize”, must be filled correctly. Otherwise,
the decoding will fail in driver side.
2.5.3. More Detail about JDEC
We had introduced two ways to decode JPEG in previous subsections. If user is expert in WMT JDEC API, he/she
may use the skill mentioned in this subsection to implement his/her application.
Get JPEG header information:
If user wants to know information about JPEG properties, it is better to call WonderMedia? API to decode header at
first.
int wmt_jdec_header (
unsigned char *buf_in,
unsigned int buf_size,
jdec_hdr_info_t *hdr);
If user knows all information about JPEG file, e.g., image width (sof_w), image height (sof_h), source color format
(4:2:0/4:2:2/4:4:4) and file size, he or she will not need to decode header again. It is better to call following one of two API
set for hardware decoder.
int wmt_jdec_set_header(jdec_hdr_info_t *hdr);
or
WonderMedia? Technologies Inc.
November 27, 2009 3 4
int wmt_jdec_set_src_color(vdo_color_fmt src_color);
int wmt_jdec_set_sof(
unsigned int sof_w,
unsigned int sof_h,
unsigned int filesize);
Prepare all necessary information:
Before starting to hardware decoder, user must set necessary setting for decoder, especially scaling mode and color
format on decode frame buffer. It is better to set decoding information at least once. Otherwise, WonderMedia? API will use
default values, which may be what user wants.
int wmt_jdec_set_decode_info(jdec_decode_info_t *di_in);
or
int wmt_jdec_set_scale(vd_scale_ratio ratio);
int wmt_jdec_set_dc_format(vdo_color_fmt decoded_color);
Allocate memory as decoded frame buffer:
Because of hardware decoder limitation, the decoded frame buffer should be physical continuous. Namely, user can
not use “malloc( )” to allocate decoded frame buffer. To allocate physical address memory in user space on Linux, we
implement MB (Memory Block) APIs to achieve it. Fortunately, ND application developers do not need to know the detail,
we encapsulate them into two APIs.
int wmt_jdec_fb_alloc (
unsigned int pic_w,
unsigned int pic_h,
vdo_color_fmt src_color,
vdo_color_fmt decoded_color_in,
jdec_input_t *arg_in);
int wmt_jdec_fb_free (jdec_frameinfo_t *info);
Once the frame buffer is useless, please do not forget to release it by calling wmt_jdec_fb_free( ).
Enable hardware decoding:
After setting header information and decoder information, user must call a “fire” wmt_jdec_attr_ready( ) function
and wmt_jdec_proc ( ) to enable hardware decoding.
int wmt_jdec_attr_ready(void);
int wmt_jdec_proc(jdec_input_t *arg_in);
Wait hardware decoding finished:
Once user enable hardware decoding, he or she must call following API to wait for decoding finish.
int wmt_jdec_finish(jdec_frameinfo_t *frameinfo);
One thing user must know is that almost time spent in this process is in wmt_jdec_finish( ). In other word, the
decoding job is finished only when wmt_jdec_finish( ) is returned instead wmt_jdec_proc ()is returned. If
performance is very critical, developers may implement their application as multi-thread, but they must consider the
overhead of context switch.
wmt_jdec_finish(frameinfo); /* thread is blocking in driver until
decoding finished */;
WonderMedia? Technologies Inc.
November 27, 2009 3 5
After wmt_jdec_finish() returned, user will get information about decoded frame buffer, which is stored in
“frameinfo”. Then user may display this frame according to “frameinfo”.
Display decoded JPEG frame buffer:
The API for user to display a frame buffer is
int wmt_jdec_show_image(
vd_handle_t *handle,
jdec_frameinfo_t *frameinfo,
jpeg_disp_mode mode);
int wmt_jdec_show(
vd_handle_t *handle,
jdec_frameinfo_t *frameinfo,
vdo_view_t *view);
Although the name of these two APIs with prefix “jdec”, it is also workable for any raw data as long as its meets
“jdec_frameinfo_t” structure. For example, user may copy a BMP file to a “jdec_frameinfo_t” structure and
then use these two APIs to display BMP file.
2.5.4. JPEG Rotation
Rotating JPEG file is necessary for almost multimedia products. WonderMedai? SoC also supports this feature.
However, “Rotation” is a feature of “GE” instead of one of video decoders. The basic concept about JPEG rotation could
be divided into stages below.
Decide the decoded dimension according to rotation angle and resolution of display device
Decode JPEG to desired dimension computed above
Allocate new frame buffer as “GE” destination buffer to save rotated image data
Use “GE” to rotate image
Free original JPEG frame buffer
Display rotated image
Free frame buffer used to save rotated image data
Sample codes below teach user how to rotate data by “GE”.
static int wmt_rotate(
vdo_framebuf_t *dst_fb,
vdo_framebuf_t *src_fb,
unsigned int rot_degree)
{
unsigned int args[6];
int fd;
int ret;
unsigned int cmd;
if( (src_fb->col_fmt != VDO_COL_FMT_ARGB) ||
(dst_fb->col_fmt != VDO_COL_FMT_ARGB)) {
DBG_ERR("Only supoprt ARGB now!\n");
return -1;
}
WonderMedia? Technologies Inc.
November 27, 2009 3 6
cmd = _IOW(0x69, 3, void *); /* GEIO_ROTATE */
args[0] = src_fb->y_addr; /* phy_src */
args[1] = dst_fb->y_addr; /* phy_dst */
args[2] = src_fb->fb_w; /* width */
args[3] = src_fb->fb_h; /* height */
args[4] = src_fb->bpp; /* bpp */
args[5] = rot_degree; /* arc */
fd = open("/dev/fb0", O_RDWR);
ret = ioctl(fd, cmd, args);
close(fd);
return ret;
} /* End of wmt_rotate() */
Currently we do not define any WMT API about rotation. Maybe we will package the sample codes above as standard
WMT API in later version.
2.5.5. PD (Partial Decoding)
There is always a tradeoff between image quality with performance. To get better quality, system needs more memory
to decode it without scaling down. In this case, the decoding performance will become worse. However, sometimes people
think image quality is most important even worse performance. In this condition, we cannot let “VDU” to scaling down
JPEG during decoding. In other words, we must let “VDU” to decode JPEG to original size. This creates a new problem. Is
the physical continuous memory always enough for hardware to decode an original picture size? For example, if there is a
JPEG with 8192x8192 dimensions, to decode it into ARGB color format, we need 8192*8192*4 = 268,435,456 = 256 MB.
It is impossible for embedded system. To solve this problem, we provide a new concept called “Partial Decoding”.
As we introduce above, to decode one JPEG frame/file by WonderMedia? hardware, user needs two memory buffers,
“Source Data Buffer” and “Destination Video FB”. We tried to use MSD technology to save size of “Source Data Buffer”.
Now we will use “PD (Partial Decoding)” to save size of “Destination Video FB”.
The basic concept of PD is that we just decode some part area which we want one time. There are two usual user
scenarios about it.
Just to show some part of JPEG on screen
For example, suppose the picture size is 4096x4096 and resolution of display device is 1024x768. If user wants to show
picture with original size on screen, i.e., only 1024x768 of 4096x4096 is displayed, we may just decode 1024x768 instead of
full picture. The memory size of “Destination Video FB” will be reduced from 64 to 3 MB in ARGB mode.
Original Size: 4096x4096x4 = 64 MB
PD Enable: 1024x768x4 = 3 MB
Usually user does want to see not only left-top of picture but also other area. Therefore, this technology of this case
may be used in “scroll” function of picture if applications have scroll bars at right and bottom sides of screen. Another case
is “zoom” function to adopt this technology.
WonderMedia? Technologies Inc.
November 27, 2009 3 7
Figure 20 – First scenario of PD
Multiple decoding JPEG strip by strip
Figure 21 – Second scenario of PD
Display area
Source size
VDU Destination Buffer
PD enabled
Final Destination Video FB
Recursive Scaling
Final display
WonderMedia? Technologies Inc.
November 27, 2009 3 8
The second scenario is used for user who wants best quality but lower performance is acceptable. Assume source size
of JPEG picture is 4096x4096 and resolution of display device is 1024x768. If we have only 3 MB physically continuous
memory buffer as “VDU Destination Buffer”, then the maximum height of strip is (3x1024x1024)/(4096x4) = 192 pixels.
So, we may decode this file 24 times (4096/192). The first decoding is from 0 ~ 191 in height, the second is 192 ~ 383….
Besides “VDU Destination Buffer”, we also need a buffer “Final Destination Video FB” (with dimension 1024x768).
The basic assumption of this way is that the quality of image with twice scaling by VDU and VPU is worse than one
with one scaling by VPU directly. It is true, if we let “VDU” to scaling image down, then image quality will lost. Of course,
“VPU” use the lost quality to scaling again, the quality must be worse. Although the quality is better, the performance must
be worse.
To learn to use PD, “ut_jdec” is a good sample. Next, we will introduce how to use PD function. Following structure
and API are used for PD only.
Structure: jdec_pd
Syntax
typedef struct {
unsigned int enable;
unsigned int x;
unsigned int y;
unsigned int w;
unsigned int h;
} jdec_pd; /* PD: Partial Decode */
Description
enable: It used to enable hardware PD (Partial Decoding) or not
x: The value means X of start decode (16xN alignment) (Unit: Pixel)
y: The value means Y of start decode (16xN alignment) (Unit: Pixel)
w: The value means Width of start decode (16xN alignment) (Unit: Pixel)
h: The value means Width of start decode (16xN alignment) (Unit: Pixel)
Note This structure is defined in WonderMedia? API (com-jdec.h).
int wmt_jdec_set_pd(vd_handle_t *handle, jdec_pd *pd_in);
/* default: Diable PD function */
Sample code below shows use PD to decode 1024x768 area of a large picture.
int main(void)
{
vd_handle_t handle;
jdec_frameinfo_t jfb;
jdec_pd pd;
/* Initialization */
wmt_init(&handle);
/* Enable PD */
pd.enable = 1;
pd.x = pd.y = 0;
pd.w = 1024;
pd.h = 768;
wmt_jdec_set_pd(&jdec_handle, &pd);
/* Start decoding if you have file path */
wmt_jdec_file(&handle, filename, &jfb);
/* Or, start decoding if you have a buffer */
WonderMedia? Technologies Inc.
November 27, 2009 3 9
/* Display the JPEG */
wmt_jdec_show_image(&handle, &jfb, JPEG_DM_BEST_FIT);
/* Free JPEG frame buffer */
wmt_jdec_fb_free(&handle, &jfb);
/* Release resource */
wmt_exit(&handle);
return 0;
}
2.6. Motion JPEG (MJPG)
Inside WonderMedia? hardware design, they are the same to decode a JEPG picture with Motion JPEG movie. They
both need “VDU” to decode a JPEG frame. In other words, developers can not decode a JEPG file and MJPEG at the same
time since there is only one hardware video decoder. Figure below shows the software architecture for JPEG and MJPEG.
WonderMedia? API
JPEG Application Media Player
(MPEG Application)
Figure 22 – Software architecture for JPEG and MJPEG
In fact, the major difference between JPEG picture and motion JPEG movie, it is in application’s behaviors. It is not
difficult to decode a JPEG file as long as following this document mentioned above. However, to play a MJPEG movie,
developer must have a “Media Player” to do more things besides decoding JPEG frame. For example, “to demux media file
header”, “to separate audio/video bitstreams”, “AV synchronization” and so on. They are all dependent on the behavior of
media player which developers use.
One thing must be kept in mind is that it is the same for WMT JDEC to decode JPEG or MJPEG. In WMT JDEC’s
point of view, it just decodes a JPEG frame. (MJPG is a set with many JPEG frames.) WMT JDEC does nothing besides
decoding JPEG frame. Other works, e.g. “AV synchronization” or “Scaling to fit screen”, should be done by media player.
It is nonsense to scaling JPEG frame when playing a movie. If developers want to scaling video data of a movie, “Real time
scaling” is the best choice instead of “VDU” scaling.
To demonstrate MJPEG, WonderMedia? formal BSP adopts “MPlayer” as the media player. (Please refer to “1.5 Open
Source Licenses” in detail about “MPlayer”) Of course, developers may use any media player as they want, but there is a
little porting effort on it. Figure below shows software architecture of MPlayer for MJPEG.
WonderMedia? Technologies Inc.
November 27, 2009 4 0
WonderMedia? API
JPEG/
MJPEG
OSS
User Space
Kernel Space
JPEG Driver
(VDU)
Audio
Driver
(Output)
FFMPEG
(Video)
FFMPEG
(Audio)
FFMPEG
Adapter
MPlayer
MPlayer
Figure 23 – Software architecture of MPlayer for MJPEG
We will not discuss too much about audio path. We just focus on JPEG path only here. If your media player supports
FFMPEG, it is an easy job to port your media player to WonderMedia? platform with hardware decoding feature because
WonderMedia? create an interface called “FFMPEG adapter” to connect WMT API with player. If not, developers must
create a new interface like “FFMPEG adapter” to connect WMT API with their player. It is case by case, we will not
discuss it in detail here.
WonderMedia? Technologies Inc.
November 27, 2009 4 1
|