WonderMedia? API Developer Guide Revision 0.3 November 27, 2009 WonderMedia? Technologies Inc. WonderMedia? Technologies Inc. November 27, 2009 i Revision History Revision Date Revision Descriptions RD 0.1 2009/10/20 Initial version Willy Chuang 0.2 2009/10/26

  • Add “1.3.4 Color Space Convert”
  • Add “2.6 Motion JPEG”
  • Correct some typos Willy Chuang 0.3 2009/11/27
  • Add “2.3.3 About VDU Scaling”
  • Add “2.3.4. Real Time and Recursive Scaling”
  • Add “2.5.5 PD (Partial Decoding)” Willy Chuang WonderMedia? Technologies Inc. November 27, 2009 ii Table of Contents REVISION HISTORY..........................................................................................................................................................I TABLE OF CONTENTS...................................................................................................................................................... II LIST OF FIGURES.............................................................................................................................................................III LIST OF TABLES................................................................................................................................................................III 1. INTRODUCTION............................................................................................................................................................ 1 1.1. PURPOSE.................................................................................................................................................................... 1 1.1.1. Abbreviation................................................................................................................................................ 1 1.2. WONDERMEDIA API CONCEPT.................................................................................................................................. 1 1.3. HARDWARE BLOCK OVERVIEW ON WONDERMEDIA SOC........................................................................................... 2 1.3.1. Video Path - Real Time Scaling ............................................................................................................ 2 1.3.2. Video Path – Recursive Scaling ............................................................................................................ 3 1.3.3. Graphic Path ............................................................................................................................................... 3 1.3.4. Color Space Convert (CSC).................................................................................................................... 4 1.4. WONDERMEDIA API RELEASE PACKAGE.................................................................................................................... 4 1.4.1. Build Application with WMT API............................................................................................................ 5 1.5. OPEN SOURCE LICENSES............................................................................................................................................ 6 1.6. WMTAPI VERSION................................................................................................................................................... 7 2. VIDEO DECODER - JPEG (WMT JDEC) .................................................................................................................... 8 2.1. OVERVIEW................................................................................................................................................................. 8 2.1.1. Class Diagram............................................................................................................................................ 8 2.1.2. Frame Buffer Structure........................................................................................................................... 9 2.1.3. Basic Concepts......................................................................................................................................... 10 2.1.4. Basic Flow and Memory Usage ........................................................................................................... 12 2.2. WMT JDECAPI INITIALIZE.................................................................................................................................... 14 2.3. DECODE JPEG......................................................................................................................................................... 15 2.3.1. APIs for Settings before Decoding .................................................................................................... 15 2.3.2. APIs for Decoding ................................................................................................................................... 19 2.3.3. About VDU Scaling ................................................................................................................................. 21 2.3.4. Real Time and Recursive Scaling....................................................................................................... 24 2.4. DISPLAY JPEG......................................................................................................................................................... 26 2.4.1. Display Mode ............................................................................................................................................ 26 2.4.2. WMT View Structure ............................................................................................................................... 28 2.5. ADVANCE JDEC FEATURES ..................................................................................................................................... 30 2.5.1. Allocate Physically Continuous Memory .......................................................................................... 30 2.5.2. MSD (Multi-Segment Decoding)......................................................................................................... 31 2.5.3. More Detail about JDEC......................................................................................................................... 33 2.5.4. JPEG Rotation........................................................................................................................................... 35 2.5.5. PD (Partial Decoding)............................................................................................................................ 36 2.6. MOTION JPEG (MJPG)............................................................................................................................................ 39 WonderMedia? Technologies Inc. November 27, 2009 ii i List of Figures FIGURE 1 –WMTAPIARCHITECTURE ......................................................................................................................................... 2 FIGURE 2 –WMTHARDWARE BLOCK ARCHITECTURE.................................................................................................................. 3 FIGURE 3 –WMTAPI FILE TREE ................................................................................................................................................. 5 FIGURE 4 –WONDERMEDIA API (HEADER FILES) HIERARCHY...................................................................................................... 5 FIGURE 5 – CLASS DIAGRAM OF WONDERMEDIA API RELATED WITH JDEC................................................................................. 8 FIGURE 6 – FRAME BUFFER ARCHITECTURE ................................................................................................................................ 12 FIGURE 7 –WONDERMEDIA BSPMEMORY MAP......................................................................................................................... 13 FIGURE 8 – DATA PATH OF HARDWARE JPEG DECODER ............................................................................................................... 14 FIGURE 9 – FLOW OF SETTING SCALING ...................................................................................................................................... 18 FIGURE 10 – DECODE TO ORIGINAL DIMENSION ........................................................................................................................... 21 FIGURE 11 – DECODE TO 1/4 DIMENSION OF ORIGINAL DIMENSION .............................................................................................. 22 FIGURE 12 – VDU SCALING ALGORITHM .................................................................................................................................... 22 FIGURE 13 – IDEA OF SCALING ALGORITHM ................................................................................................................................ 23 FIGURE 14 – EXAMPLE OF WMT_JDEC_FILE_IN_SIZE() ................................................................................................................ 24 FIGURE 15 – FLOW OF REAL TIME SCALING................................................................................................................................. 24 FIGURE 16 – FLOW OF RECURSIVE SCALING + REAL TIME SCALING.............................................................................................. 25 FIGURE 17 – DISPLAY MODE: IF PICTURE IS SMALLER THAN DISPLAY AREA.................................................................................. 27 FIGURE 18 – DISPLAY MODE: IF PICTURE IS LARGER THAN DISPLAY AREA.................................................................................... 28 FIGURE 19 – VDO_VIEW_T INTRODUCTION ................................................................................................................................. 29 FIGURE 20 – FIRST SCENARIO OF PD........................................................................................................................................... 37 FIGURE 21 – SECOND SCENARIO OF PD....................................................................................................................................... 37 FIGURE 22 – SOFTWARE ARCHITECTURE FOR JPEG AND MJPEG ................................................................................................ 39 FIGURE 23 – SOFTWARE ARCHITECTURE OF MPLAYER FOR MJPEG............................................................................................ 40 List of Tables TABLE 1 – OPEN SOURCE USED IN LINUX BSP .............................................................................................................................. 6 TABLE 2 – RELATIONSHIP BETWEEN SOURCES WITH DECODED COLORS ....................................................................................... 17 WonderMedia? Technologies Inc. November 27, 2009 1 1. Introduction 1.1. Purpose This document provides a concept of idea for customers to develop application on Linux operation system based on WonderMedia? SoC platform. For this purpose, customer should understand WonderMedia? APIs framework and interface in detail. The interface includes video decoder, e.g., JPEG, video post processing, graphic acceleration, security engine (cipher) (option) and so on. The contents of this document are suitable for all WonderMedia? SoC named WM8xxx series in theory. However, they are not suitable for old VT8xxx SoC. 1.1.1. Abbreviation BSP Board Support Package CSC Color Space Convert EVB EValuation Board FB Frame Buffer, a physically continuous memory buffer used to store video data GE Graphic Engine GOV Graphic OVerlay GOVR Graphic OVerlay (Read function) GOVW Graphic OVerlay (Write function) IDE Integrated Development Environment JDEC Jpeg DECoder HW Hardware MB Memory Block, a Linux driver for application to access physically continuous memory buffer MSD Multi-Segment Decoding MVL MontaVista? Linux® PD Partial Decoding SDK Software Development Toolkit SE Security Engine SW Software VDMA Video DMA, a hardware block in WonderMedia? Soc VDU Video Decode Unit, a hardware block in WonderMedia? SoC VPP Video Post Processing function block VPU Video Post Unit, a hardware block in WonderMedia? Soc WMT WonderMedia? Technologies 1.2. WonderMedia? API Concept WonderMedia? API, “WMT API”, is a set of functions provided by WonderMedia? for application developers, who would develop their product on WonderMedia? platform. Basically WMT APIs access almost hardware blocks, which are related with multimedia, though Linux drivers. It is very important for customers to be proficient in WMT API if they must develop applications on WonderMedia? platform. Roughly we may divide WMT APIs into some groups. They are “SE”, “VDU”, “GE”, “MB”, “VDMA” and “VPP”. “VPP” is an abbreviation of “Video Post Processing”. It has not a real hardware block. In generally, VPP we called contains “VPU”, “GOV (R/W)” and “SCALER” hardware blocks. “VPU” is a hardware block used for further scaling up/down images in frame buffers and send them “GOV (R/W)” blocks. WonderMedia? Technologies Inc. November 27, 2009 2 “VDU” means “Video Decode Unit”. It is used to decode video bitstreams, (e.g., JPEG, MPEG, H.264 and so on), which depends on SoC capabilities. The output data of VDU should be saved on a physically continuous memory buffer, named “video frame buffer”. We also call this “video frame buffer” generated by VDU as “video plane”. “GOV” is an important hardware block for graphic overlay and alpha mixing. In some WonderMedia? Soc, there are two blocks called “GOVW” and “GOVR” for “write” and “read”, respectively. “GOVW” mixes graphic plane and video planes with alpha blending and write the result to DRAM. Then “GOVR” reads the data on DRAM and send it to output device. “GE” means “Graphic Engine”. It is used to handle graphic acceleration by hardware. We call the plane operated by GE as “graphic plane”. “Cipher” or “SE” (Security Engine) is a hardware block, which customer may use to encrypt or decrypt data. “VDMA” is a hardware block used to copy or “bitblt” video data. “MB” means “Memory Block”, which is a software block (Linux driver) used to allocate physically continuous memory buffer for user-space applications. WonderMedia? API Linux Application JPEG/ MJPEG Security Engine Graphic VPP Engine User Space Kernel Space JPEG Driver Cipher Driver GE Driver VPP Driver Figure 1 – WMT API Architecture 1.3. Hardware Block Overview on WonderMedia? SoC In this subsection, we will introduce WonderMedia? Soc hardware block architecture. One thing reader must keep in mind is that this figure below is roughly overview and there may be a little difference between different WonderMedia? Soc. 1.3.1. Video Path - Real Time Scaling WonderMedia? Technologies Inc. November 27, 2009 3 Figure 2 below shows a simple user scenario about video and graphic data path. Application must allocate “Video FB#1” and inform “VDU” and “VDU” will render decoded video data on it. Then application “sends” “Video FB#1” to “VPU”. In property timing, “GOVW” will overlay “Graphic FB” and “Video FB#1” and copy them to “Video FB#3” or “Video FB#4” automatically. Sometimes, dimension of “Video FB#1” is difference with “Video FB#3” because of scaling by “VPU”. In this scenario, hardware scaling is finished by “VPU” and “GOVW” automatically. We call it as “Real time scaling”. 1.3.2. Video Path – Recursive Scaling In Figure 2 below, if application does not send “Video FB#1” to “VPU” directly. Instead, it sends “Video FB#1” (source buffer) and “Video FB#2” (destination buffer) to “SCALER”. Then “SCALER” scales “Video FB#1” onto “Video FB#2”. (LOOK OUT: These two frame buffers were allocated by application in advance.) After that, application “sends” “Video FB#2” to “VPU” and “VPU” and “GOVW” “copy” “Video FB#2” to “Video FB#3” without scaling by hardware automatically. In this scenario, hardware scaling is finished by “SCLAER”. We call it as “Recursive scaling”. GE VPU SCLAER GOVW GOVR Output Device System Memory Input VDU Device Video FB #1 Video FB #2 Grapihc FB Video FB #4 Video FB #3 Recursive Scaling Real Time Scaling Figure 2 – WMT Hardware Block Architecture In software’s point of view, the “Video FB#1” and “Video FB#2” must be allocated in user space, including applications or WMT API. The “Video FB#3” or “Video FB#4” are allocated by Linux driver in kernel space. As for “Graphic FB”, it is pre-allocated before system was booted. Moreover, all frame buffers above should be physically continuous memory since they are accessed by hardware block. It is important to know that the data formats of video FB supported by WonderMedia? hardware design are “YUV4:2:0”, “YUV4:2:2H”, “YUV4:4:4” and “ARGB” only. In other word, if user copy or move data to frame buffer by other format, the result will be correct because hardware cannot recognizes it. 1.3.3. Graphic Path In software engineer’s point of view, application should be access “Graphic FB” directly. In general user scenario, “Graphic FB” is always access by popular graphic libraries, e.g., DirectFB®. And application calls APIs of popular graphic libraries. Current graphic libraries which WonderMedia? adopt in Linux BSP is “GTK+/DirectFB®” and “VGUI”, which is proprietary library developed by WonderMedia?. (You may get more information about DirectFB® or GTK+ from hyperlink below) DirectFB stands for Direct Frame Buffer. It is a software library for the GNU/Linux operating system that provides "hardware graphics acceleration, input device handling and abstraction, integrated windowing system with support for translucent windows and multiple display layers, not only on top of the Linux Framebuffer Device." It is very popular for Linux operating system and open source, we adopt it as our graphic library in SDK to handle graphic effect. Besides, there WonderMedia? Technologies Inc. November 27, 2009 4 are many tutorial documents or articles about DirectFB and user can study or get any information, which he/she wants, easily. (For example, refer to http://www.directfb.org/docs/DirectFB_Reference/ ). GTK+ is a highly usable, feature rich toolkit for creating graphical user interfaces which boasts cross platform compatibility and an easy to use API. (Please refer to http://www.gtk.org/ ) 1.3.4. Color Space Convert (CSC) There are two hardware blocks have CSC capabilities inside, they are “VDU” and “GOVR”. “VDU” has only one CSC function (from YUV to RGB in JFIF algorithm). But the “GOVR” have two CSC functions (from YUV to RGB or RGB to YUV in some algorithms). In theory, the CSC algorithm in “VDU” and “GOVR” in hardware design should be the same. If developers want to verify video quality by dumping video data directly, they may let “VDU” to decode into ARGB mode (“Video FB #1”) and let “VPU” bypass to “GOVR” buffer (“Video FB#3”) (Note: bypass means no scaling in “VPU”). In this case, “Video FB

    #1” and “Video FB #3” are the same. Then developers may dump “Video FB#1” by “ut_jdec” or “Video FB #3” by “ut_vpp”, respectively. As for how to dump it by unit test programs, please refer to another document. 1.4. WonderMedia? API Release Package Formal WMT API in WonderMedia? Linux BSP release package includes following components. |-- wmt_api\ |-- release\ |-- include\ |-- com-jdec.h (file) |-- com-mb.h (file) |-- common-api.h (file) |-- com-se.h (file) |-- com-vd.h (file) |-- com-vdma.h (file) |-- com-video.h (file) |-- wmt-api.h (file) |-- wmt-jdec.h (file) |-- wmt-mb.h (file) |-- wmt-se.h (file) |-- wmt-vd.h (file) |-- wmt-vdma.h (file) |-- wmt-vpp.h (file) |-- lib\ |-- libwmtapi.a (file) |-- libwmtapi.so (file) |-- libwmtapi.so.1.0.1 (file) WonderMedia? Technologies Inc. November 27, 2009 5 Figure 3 – WMT API File Tree The WonderMedia? API package may shorten as |-- release |-- include\ |-- com-xxx.h (files) |-- wmt-xxx.h (files) |-- lib\ |-- libwmtapi.a (file) |-- libwmtapi.so (file) Here, libwmtapi.a is used for static link and libwmtapi.so is used in shared library. The files with prefix “com-” mean they are used by both user space (API) and kernel space (Linux drivers). And the files with prefix “wmt-“ means these header files are used by applications in user space only. Following figure describes the relationship between “com-xxx.h” with “wmt-xxx.h”. com-video.h (vdo_ ) com-vd.h (vd_ ) com-jdec.h (jdec_ ) com-mpeg.h com-vpp.h (vpp_ ) Only define following structures vdo_framebuf_t vdo_view_t Basic "class" file for all video decoders wmt-jdec.h (wmt_jdec_ ) Files above this line are used for both kernel and user spaces Files below this line are used for user space only nd_api.h (nd_ ) wmt-vpp.h (wmt_vpp_ ) com-ge.h (ge_ ) wmt-se.h (wmt_se_ ) wmt-ge.h (wmt_ge_ ) com-se.h (se_ ) WMT API

    #include(): Usage: (a-page-name-you-want-to-include[,title|,notitle])
    Figure 4 –WonderMedia? API (Header Files) Hierarchy 1.4.1. Build Application with WMT API To build applications, which are related with WMT API, user must have a correct setting of “wmt_api”. The related paths of developer’s environment may be different for everybody. For example, assume your application is “ND SDK”, which folder is “nd_sdk_2.2.0_090618”. To build it successfully, user must confirm whether the paths in Makefile are correct or not. API_PATH = ../../wmt_api/release/ WonderMedia? Technologies Inc. November 27, 2009 6 ARM_PATH = /opt/arm-linux The environment of WonderMedia? API for this example should be set as below. |-- nd_sdk\ |-- nd_sdk_2.2.0_090618\ |-- Makefile (file) |-- wmt_api\ |-- release\ 1.5. Open Source Licenses Reader may get more information about these libraries from hyperlink below. All source codes in WMT API are proprietary and developed by WonderMedia?. These codes provide an interface for application to access WonderMedia? hardware blocks through Linux driver. Moreover, the APIs is independent with application or Linux drivers. It is an independent library. Therefore, it is not open source. However, there are many kinds of multimedia SoC provided by WonderMedia?. The following table shows open source licenses, which libraries may be used in WonderMedia? Linux SDK for demonstration purpose. Customers need to understand these open source licenses if they want to adopt them to develop their application and products. Module License GTK+ LGPL DirectFB LGPL MPlayer GPL FFMPEG LGPL Linux kernel (MVL 5.0) GPL + MVL Table 1 – Open Source Used in Linux BSP Reader may get more information about these libraries from hyperlink below. MPlayer is a free and open source media player distributed under the GNU General Public License. The program is available for all major operating systems, including Linux and other OS. MPlayer supports a wide variety of media formats. In addition to its wide range of supported formats MPlayer can also save all streamed content to a file. (Please refer to MPlayer website http://www.mplayerhq.hu/ ) FFmpeg is a complete solution to record, convert and stream audio and video. It includes libavcodec, the leading audio/video codec library. FFmpeg is developed under Linux, but it can be compiled under most operating systems, including Windows. (Please refer to http://ffmpeg.mplayerhq.hu/ ) DirectFB stands for Direct Frame Buffer. It is a software library for the GNU/Linux operating system that provides "hardware graphics acceleration, input device handling and abstraction, integrated windowing system with support for translucent windows and multiple display layers, not only on top of the Linux Framebuffer Device." It is very popular for Linux operating system and open source, we adopt it as our graphic library in SDK to handle graphic effect. Besides, there are many tutorial documents or articles about DirectFB and user can study or get any information, which he/she wants, easily. (For example, refer to http://www.directfb.org/docs/DirectFB_Reference/ ). GTK+ is a highly usable, feature rich toolkit for creating graphical user interfaces which boasts cross platform compatibility and an easy to use API. (Please refer to http://www.gtk.org/ ) WonderMedia? Technologies Inc. November 27, 2009 7 1.6. WMT API Version Current latest version of WMT API is “4.0.3”. If user’s application is limitation some specified version of WMT API, it is better to call the version-check API below to avoid some trouble. For example, if your application requires WMT API “4.0.0” or later version, it is better to add sample codes below in your application. if( wmt_api_ver_require("4.0.0") ){ return -1; } To understand detail release notes, user may refer to “wmt-api.h”, which is under “/release/include/” of WMT API. WonderMedia? Technologies Inc. November 27, 2009 8 2. Video Decoder - JPEG (WMT JDEC) In this chapter, we will mainly introduce how to use WMT API (JPEG decoding related). 2.1. Overview 2.1.1. Class Diagram Figure 5 – Class Diagram of WonderMedia? API related with JDEC In applications developer’s point of view, the most important “class” in figure above is “jdec_frameinfo_t”. Almost application would use or operate this structure. Sometimes, “jdec_hdr_info_” may be used by some applications. As for other structures, they are always used inside WMT JDEC API. Application developer may ignore them at first stage. WonderMedia? Technologies Inc. November 27, 2009 9 2.1.2. Frame Buffer Structure The structure “vdo_framebuf_t” is the core structure in all video related API. Developer must know this structure in detail. Structure: vdo_framebuf_t Syntax typedef struct { unsigned int y_addr; unsigned int c_addr; unsigned int y_size; unsigned int c_size; unsigned int img_w; unsigned int img_h; unsigned int fb_w; unsigned int fb_h; unsigned int bpp; vdo_color_fmt col_fmt; unsigned int h_crop; unsigned int v_crop; } vdo_framebuf_t; Description This structure describes all settings of one frame buffer y_addr: physical address of this frame buffer for Y (or RGB) plane y_size: buffer size of this frame buffer for Y (or RGB) plane c_addr: physical address of this frame buffer for C plane c_size: buffer size of this frame buffer for C plane img_w: valid picture width on this frame buffer for both Y (or RGB) /C planes img_h: valid picture height on this frame buffer for both Y (or RGB) /C planes fb_w: the scan line offset of this frame buffer in width for both Y(or RGB) /C planes fb_h: the lines of this frame buffer in height for both Y(or RGB) /C planes bpp: bits per pixel on Y (or RGB)/C plane col_fmt: color format on Y (or RGB)/C plane h_crop: cropping on X coordinate v_crop: cropping on Y coordinate Note If the color format of this frame buffer is RGB, c_addr and c_size would be both zero. And y_addr and y_size will indicate the address and buffer size of RGB plane, respectively. Following figure shows the concepts of some members in this structure. y_addr or c_addy points to left-top. WonderMedia? Technologies Inc. November 27, 2009 1 0 In almost case, h_crop and v_crop are usually zero. This structure is defined in WMT API (com-video.h). For application, this structure “vdo_framebuf_t” is not enough since there is no user space address. Application cannot do anything with this structure. So we create a new structure called “jdec_frameinfo_t”, which is a “subclass” of “vdo_framebuf_t” and used in almost WMT JDEC API. Structure: jdec_frameinfo_t Syntax typedef struct { vdo_framebuf_t fb; unsigned int y_addr_user; unsigned int c_addr_user; unsigned int scaled; } jdec_frameinfo_t; Description This structure describes all settings of one frame buffer used for JPEG driver only. This structure should be thought as a “subclass” of vdo_framebuf_t class. fb: For more detail, please refer to definition in vdo_framebuf_t. y_addr_user: logical address of this frame buffer for Y (or RGB) plane in user space c_addr_user: logical address of this frame buffer for C plane in user space scaled: means the image on this frame buffer whether was scaled by JPEG decode driver. (0: original size; 1: scaled) (Please do NOT modify this value in application) Note This structure is defined in and shared with WMT API (com-jdec.h) instead of ND API. 2.1.3.Basic Concepts To be proficient in WMT JPEG API, some basic concepts about decoding JPEG are necessary for customers. Profile According to JPEG specification in “ISO/IEC 10918-1”, there are many kinds of profiles. However, WonderMedia? only supports “Baseline DCT” profile hardware decoding. (Symbol: SOF0 and Marker: 0xFF 0xC0) If customers try to use WMT API to decode non-baseline JPEG file, they may get following error message. WonderMedia? Technologies Inc. November 27, 2009 1 1
  • The profile of JPEG does not be supported SOLUTION: Please use software JPEG decoder to decode this file Resolution They are some kinds of definitions about “resolution”. To avoid confusion, we define them as Source resolution: It means the resolution of source pictures. For example, the source resolution of one JPEG file with dimension 600x400 is 600x400. Decoded resolution: It means resolution of the picture decoded by hardware. In theory, it should be equal to source resolution, but they may be different if customers request decoder to scale it down. WonderMedia? JPEG hardware decoder may decode a JPEG file to its original, 1/2, 1/4 or 1/8 size. Frame buffer resolution: It means resolution of a memory buffer, which stores decoded JPEG. Because of hardware alignment limitation, it should be closed to but larger than decoded resolution. Display resolution: It means output resolution of screen. In theory, it should not be any arbitrary value. It must fit VESA specification, e.g., “XGA (1024x768)” or “SVGA (800x600)”. There no certain relationships between these four resolutions. It depends on customer’s applications. Color Space/Color Format The term “color space or color format” means horizontal and vertical sampling in a JPEG file, e.g., YUV 4:4:4 or 4:2:2H. In general speaking, WonderMedia? support all color space in JPEG source file. They include YUV 4:2:0, 4:1:1, 4:2:2H, 4:2:2V; 4:4:4 and gray level. However, the decoded color space of WonderMedia? hardware decoder only supports YUV4:2:0, YUV4:2:2H, YUV4:4:4 or RGB 32 bit modes. Source color space and decoded color space may be different. Customers may use some WMT API to do that if necessary. Scaling They are some kinds of definitions about “scaling”. To avoid confusion, we define them as Hardware decoding scaling: It means the scaling function of WonderMedia? JPEG hardware decoder. (Only 1/1, 1/2, 1/4 and 1/8 are supported). Hardware scaler scaling: It means the scaling functions of WonderMedia? SCL444 hardware block. It is called by “recursive scaling”. VPP scaler scaling: It means the scaling functions of WonderMedia? VPU hardware block. It is called by “real time scaling”. About “recursive scaling” and “real time scaling”, please refer to “subsection 1.3”. Frame Buffer The term “Frame Buffer (FB)” here means a memory buffer with continuous physical address and it is used to store decoded JPEG data or other video data. Basically, frame buffer is almost used by hardware components. Therefore there is alignment limitation in line size (or line width). However, it should be transparent for application developer. In WMT JDEC API, there are two main color spaces, i.e., YUV and RGB. For YUV, WMT JDEC API allocates two frame buffers for one picture. They are called “Y-Plane”, which saves Y data (luminance), and “C-Plane”, which stores Cb WonderMedia? Technologies Inc. November 27, 2009 1 2 and Cr data (chrominance). The bytes per line of Y-plane and C-plane are the same with source width of picture if it satisfies hardware limitation (64 bytes alignment). As for the bytes per line of RGB plane, it is four times of source width of picture (64 bytes alignment). For example, if source picture is 1000x480, then decoded frame buffer should be like one of two cases below. Y-Plane Y0 Y1 Y2 Y3 C-Plane Cb Cr RGB-Plane B G R A 1024 bytes 4032 bytes 1000 bytes 4000 bytes Figure 6 – Frame buffer architecture One thing which developer should know is the format of ARGB. It is 32 bits and offset 0 is blue (B), offset 1 is green (G), offset 2 is red (R) and offset 3 is alpha value (A), which is not used for JPEG. (The order is similar with Microsoft BMP format in 32 bits mode). When developers should use YUV or RGB mode, it depends on their application. Please keep in mind that application has better performance, lower memory bandwidth and good quality in YUV mode if the data source is originally in YUV mode, e.g., JPEG file. 2.1.4. Basic Flow and Memory Usage Generally speaking, there are standard steps to finish a JPEG decoding job. They are respectively Allocate a buffer to save source data called “source data buffer”, which may be NOT physically continuous Do correct settings, e.g., decoded color, scaling… Allocate a frame buffer to save destination data called “destination video frame buffer”, which should be physically continuous Start hardware JPEG decoding to (1/1, 1/2, 1/4 or 1/8) Scaling by “Recursive Scaling Mode” to desired resolution (Option) Rotate video frame buffer (Option) Display this picture if this decoding is successful The order of these steps is the same with application order to decode JPEG. Allocate a buffer to save source data: Suppose the resolution of source image file, which user wants to decode, is 4096x4096 in YUV4:4:4 mode. The size without compression is about 4096x4096x3 = 50,331,648 = 48 MB. Assume the compression ratio is 50%, we need 24 MB to store this compressed file. So the size of “source data buffer” is 24 MB. The pseudo code in your application is filesize = 25165824; /* 24 MB */ fp = fopen(“test.jpg”, "rb"); /* Open image file */ WonderMedia? Technologies Inc. November 27, 2009 1 3 src_buf = malloc(filesize); /* Allocate source data buffer */ fread(src_bu, 1, filesize, fp); /* Copy data to src_buf */ However, 24 MB is not easy to be allocated in embedded system. To guarantee the decoding process is successful if system memory is shortage, we provide a method called “MSD (Multi-Segment Decoding). We will introduce MSD in detail later. Do correct settings: Important settings about JPEG decoding are “decoded color” and “scaling ratio”. They will impact the memory size of “destination video frame buffer”. We will introduce it in subsection below later. Allocate a frame buffer to save destination data: Since frame buffer to save destination data, i.e., “destination video frame buffer”, should be physically continuous, developers cannot guarantee they may always allocate it each time. In WMT API architecture, application must allocate physically continuous memory by MB (Memory Block) driver. The video frame buffer is configured in Linux Menuconfig. (Default is 32 MB) Because DRAM on platform is limited, developers cannot decode too large JPEG file, though the upper limitation of WonderMedia? hardware JPEG is 65536x65536. Pre-allocated Memory Memory Pool For Linux kernel & application Allocated Memory by MB Figure 7 – WonderMedia? BSP Memory Map We could divide total memory (DRAM) into three parts, “Pre-allocated Memory”, “Allocated memory by MB” and “Memory Pool for Linux kernel and application”. “Pre-allocated Memory” is mainly used for frame buffer device driver, i.e., graphic layer. “Allocated memory by MB” is mainly used for both video decoders and VPP. If there is no any display device on product, these two blocks above are not necessary. The rest memory is used for Linux kernel and applications. Develops must fully understand how many memory their application need on their product. Follow the example above. If user wants to decode this file as original size in ARGB mode, then the frame buffer size needed is 4096x4096x4 = 67,108,864 = 64 MB. This decoding will be fail if there is no 64 MB physically continuous memory. However, in most embedded products in current markets, the resolution of output device is smaller than full HD (1920x1080). So we may let JPEG hardware decoder to scale this picture down to 2048x2048 automatically by doing correct settings. Then the video frame buffer size required will be reduced to 2048x2048x4 = 16,777,216 = 16 MB. 16 MB is more possible to be allocated successfully in embedded produce than 64 MB. The key concept is the maximum decoded resolution is dependent on MB buffer size configured in Linux kernel. The more MB buffer size you reserve, the bigger decoded resolution is. Understand this concept is very important for developers. After all, embedded product is not personal computer. The cost of DRAM is also a considered factor. WonderMedia? Technologies Inc. November 27, 2009 1 4 Figure 8 – Data Path of hardware JPEG decoder Start hardware JPEG decoding: There are some WMT JDEC APIs for application to decode JPEG. We will introduce it in subsection below later. Scaling by “Recursive Scaling Mode” to desired resolution: From Figure 7 above, application must allocate another video frame buffer, e.g., “Destination Video FB #2”, as destination buffer of “SCALER”. In most case, the size of “Video FB #2” is smaller than “Video FB #1”. (For example, the resolution of “Video FB #1” may be 2048x2048, but “Video FB #2” is only “1024x768”.) After scaling finished, “Video FB

    #1” may be released immediately. This step is option. Not all application needs this step. Rotate video frame buffer: This step is option. Not all application needs this step. We will introduce it in subsection below later. Display this picture: There are some WMT JDEC APIs for application to display a JPEG frame buffer. We will introduce it in subsection below later. 2.2. WMT JDEC API Initialize In this subsection, we will discuss about how to initialize drivers if we want to use hardware JPEG decoder. In WMT API architecture, to use hardware JPEG decoder, user must initialize VPP (video post process) and VD (common video decoders) drivers. Following is easiest sample codes to initialize/release drivers for JPEG decoders.

    #include "wmt-vpp.h"

    #include "wmt-vd.h"

    #include "wmt-jdec.h" int wmt_init(vd_handle_t *handle) { wmt_vpp_init(); if( wmt_vd_init(handle, VD_JPEG) != 0) { printf("*E* Video Decoder Initial FAIL!\n"); return -1; } wmt_vpp_govw_set_ge_visible(0); /* Turn off GE layer */ WonderMedia? Technologies Inc. November 27, 2009 1 5 wmt_vpp_govw_set_vpu_visible(1); /* Turn on VPP layer */ return 0; } /* End of wmt_init() */ int wmt_exit(vd_handle_t *handle) { wmt_vpp_govw_set_ge_visible(1); /* Turn on GE layer */ wmt_vpp_govw_set_vpu_visible(0); /* Turn off VPP layer */ wmt_vd_exit(handle); wmt_vpp_exit(); return 0; } /* End of wmt_exit() */ int main(void) { vd_handle_t jpeg_handle; wmt_init(&jpeg_handle); wmt_exit(&jpeg_handle); return 0; } In WMT API architecture, JPEG decoded frame buffer is put into video path, i.e., this frame buffer will be as input buffer of VPU instead of GE. So, to see image in video path, we must turn on VPU and turn off GE. And we must turn off VPU and turn on GE to return to default status. In almost application, default case is enable graphic layer and disable video layer. LOOK OUT: If user turns on both graphic and video layer, he/she must take care of overlay settings carefully. We will introduce this topic about overlay in another chapter. The behavior of “vd_handle_t” is similar with standard I/O API. User must keep this handle until it is released because it will be used in each WMT JDEC APIs. For multi-thread applications, each thread should initialize and release drivers and own itself handle. In the sample codes above, reader may find that there are three header files are included, “wmt-vpp.h”, “wmt-vd.h” and “wmt-jdec.h” “wmt-vpp.h” is necessary for WMP VPP API if developers want to display frame buffer. “wmt-vd.h” is common video decoders. It is necessary for JPEG decoder since JPEG decoder is a “subclass” of command VD (video decoder). “wmt_jdec.h” is necessary for application to use WMT JPEG API. 2.3. Decode JPEG We had discussed the seven steps about JPEG decoding in previous subsection briefly. In this subsection, we will introduce the steps 2 ~ 4 in detail. 2.3.1. APIs for Settings before Decoding When user wants to decode a JPEG file, he/she must do correct settings. If he/she will not do any settings, API inside will use default value. However, default value may be not what user wants in some cases. So we recommend that application developer always call settings APIs to set correct ones. WonderMedia? Technologies Inc. November 27, 2009 1 6 Decoded Color In theory, user may set decoded color as one of enumeration “vdo_color_fmt”. However, there are two kinds of decoded color which are recommended, they are “VDO_COL_FMT_AUTO” and “VDO_COL_FMT_ARGB”. If application does not allocate the “destination video frame buffer”, e.g., allocated by API automatically, it may not care it too much. Otherwise, application must care about bpp (bits per pixel) is different with each color format. For instance, bpp is 4 for ARGB; 3 for YUV4:4:4; 2 for YUV4:2:2H and 1.5 for YUV4:2:0. Enum: vdo_color_fmt Syntax typedef enum { VDO_COL_FMT_YUV420, VDO_COL_FMT_YUV422H, VDO_COL_FMT_YUV422V, VDO_COL_FMT_YUV444, VDO_COL_FMT_YUV411, VDO_COL_FMT_GRAY, VDO_COL_FMT_ARGB, VDO_COL_FMT_AUTO, VDO_COL_FMT_MAX, VDO_COL_FMT_UNKNOWN } vdo_color_fmt; Description This enum is used and shared for all WonderMedia? APIs to set color format on a frame buffer. YUV color format (VDO_COL_FMT_YUV420, VDO_COL_FMT_YUV422H, VDO_COL_FMT_YUV422V, VDO_COL_FMT_YUV444, VDO_COL_FMT_YUV411): For all YUV color format, we will use 2 planes layout, i.e., Y and C planes. Gray level color format (VDO_COL_FMT_GRAY): Although it is gray level, but we still allocate a C plane with full 0x80 data on it because of our hardware limitation. RGB color format (VDO_COL_FMT_ARGB): For RGB space, we only support RGB 32 bits, i.e., ARGB where A is a dummy byte. VDO_COL_FMT_AUTO: The value is used for JPEG only. For WonderMedia? SoC, JPEG hardware decoder may decode a JPEG file to YUV 420, 422H, 444 or ARGB color format for different source format. The best way to set decoded color format is set to VDO_COL_FMT_AUTO or VDO_COL_FMT_ARGB. In auto mode, JPEG decoder driver will decode YUV 420, 411 source to 420, YUV 444 source to 444. Other source color formats will be decoded to YUV 422H. In RGB mode, JPEG decoder driver will decode it to RGB 32 bits no matter what kind of source format. Note This enumeration is defined in WonderMedia? API (com-video.h). Application may call the API below to set decoded color. int wmt_jdec_set_dc_format( vd_handle_t *handle, vdo_color_fmt decoded_color) Currently WonderMedia? hardware decoder only supports 3 components (YUV) in JPEG file (RGB or CMYK components are not supported). Therefore, we assume the source format of JPEG file is always in YUV domain. So if user set decoded color as “VDO_COL_FMT_AUTO”, the JPEG driver will decode JPEG to YUV. If user wants data in RGB domain, please set “VDO_COL_FMT_ARGB”. Source color format Set Decoded Color Result YUV 4:2:0/4:1:1 VDO_COL_FMT_AUTO VDO_COL_FMT_YUV420 WonderMedia? Technologies Inc. November 27, 2009 1 7 VDO_COL_FMT_ARGB VDO_COL_FMT_ARGB YUV 4:2:2H/4:2:2V VDO_COL_FMT_AUTO VDO_COL_FMT_YUV422H VDO_COL_FMT_ARGB VDO_COL_FMT_ARGB YUV 4:4:4 VDO_COL_FMT_AUTO VDO_COL_FMT_YUV444 VDO_COL_FMT_ARGB VDO_COL_FMT_ARGB Others Not Supported Not Supported Table 2 – Relationship between sources with decoded colors JPEG Decoder Scaling As we talk above, WonderMedia? JPEG hardware decoder support scaling down feature (original, 1/2, 1/4 and 1/8). To save memory, let JPEG hardware decoder to decode and scale down together is helpful. Enum: vd_scale_ratio Syntax typedef enum { SCALE_ORIGINAL, SCALE_HALF, SCALE_QUARTER, SCALE_EIGHTH, SCALE_BEST_FIT, SCALE_THUMBS, SCALE_MAX } vd_scale_ratio; Description This enum is used and shared for all WMT JDEC APIs to set scaling ratio for hardware decoder. SCALE_ORIGINAL: Do not scale source and keep its original size SCALE_HALF: Let hardware decoder scale down JPEG to 1/2 size in both width and height. SCALE_QUARTER: Let hardware decoder scale down JPEG to 1/4 size in both width and height. SCALE_EIGHTH: Let hardware decoder scale down JPEG to 1/8 size in both width and height. SCALE_BEST_FIT: Let WMT JDEC API decide the scaling ratio according to source size and resolution of display device. SCALE_THUMBS: It is a customize mode. User may input any value as width and height. Note This enumeration is defined in WonderMedia? API (com-jdec.h). Application may call the API below to set scaling ratio. int wmt_jdec_set_scale(vd_handle_t *handle, vd_scale_ratio ratio) Although there are six choices, which user may set, in this API, only four values of these six are set to hardware registers. User must keep one thing in mind that the decoded frame buffer size is only 1/1, 1/2, 1/4 or 1/8 even he/she chooses “Best Fit” or “Thumbs”. WonderMedia? Technologies Inc. November 27, 2009 1 8 Application WMT JDEC API JDEC Driver HW JPEG Decoder SCALE_ORIGINAL/ SCALE_HALF/ SCALE_QUARTER/ SCALE_EIGHTH/ SCALE_BEST_FIT/ SCALE_THUMBS SCALE_ORIGINAL/ SCALE_HALF/ SCALE_QUARTER/ SCALE_EIGHTH/ Figure 9 – Flow of setting scaling For example, assume source JPEG size is 4000x4000 and resolution of display device is “1024x768”. If user set scaling mode as “Best Fit”, hardware JPEG decoder may decodes it to 1000x1000 or 2000x2000 according to scaling algorithm, which user set or default value in WMT JDEC API. Enum: jpeg_scale_alg Syntax typedef enum { JPEG_SA_SMART, JPEG_SA_QUALITY, JPEG_SA_FAST, JPEG_SA_MAX } jpeg_scale_alg; /* SA = Scale Algorithm */ Description This enum is used and shared for all WMT JDEC APIs to set scaling algorithm. JPEG_SA_SMART: Use JPEG_SA_QUALITY or JPEG_SA_FAST automatically according to picture size and requested size JPEG_SA_QUALITY: Decode to a larger size than requested size by HW decoder. Then use HW SCALER to scaling down to requested size (better quality). JPEG_SA_FAST: Decode to a smaller size than requested size by HW decoder. Then use HW SCALER to scaling up to requested size (better performance). Note This enumeration is defined in WonderMedia? API (wmt-jdec.h). The default value of scaling algorithm in WMT JDEC API is “JPEG_SA_SMART”. For example, in the example above, WMT API request hardware to decode source to 1000x1000 because the smart algorithm decides scaling ratio by “Distance”. Distance[ (1000x1000) – (1024x768) ] < Distance[(2000x2000) – (1024x768)] If users do not want to be affected by this uncertain factor, they must set scaling ratio as 1/1, 1/2, 1/4 or 1/8 correctly in wmt_jdec_set_scale( ). Get JPEG Information WonderMedia? Technologies Inc. November 27, 2009 1 9 Sometimes it is useful to know header information of JPEG before starting to decode JPEG. For example, if application does not know picture resolution and resolution of display device, it cannot know how to set scaling ratio. Following WMT JDEC API is used for application to parse JPEG header. int wmt_jdec_header ( unsigned char *buf_in, unsigned int buf_size, jdec_hdr_info_t *hdr); Structure: jdec_hdr_info_t Syntax typedef enum { JT_FRAME, JT_FIELD_TOP, JT_FIELD_BOTTOM } jpeg_type; typedef struct { jpeg_type type; int field_size; } avi1_t; typedef struct { unsigned int profile; unsigned int sof_w; unsigned int sof_h; unsigned int filesize; vdo_color_fmt src_color; avi1_t avi1 } jdec_hdr_info_t; Description profile: It means the JPEG file coding profile. Only Baseline DCT (0xFF 0xC0) is supported by WonderMedia? JPEG decoder sof_w: The value means valid image width in JPEG header. sof_h: The value means valid image height in JPEG header. filesize: The value means the file size (in bytes) of JPEG file. src_color: It means the color format in JPEG header. avil: The value means “avi1” in APP. It used for MJPEG only. In some MJPEG, one JPEG frame is divided into two fields. We need this information when decode this kind of frame. Note This structure is defined in WonderMedia? API (com-jdec.h). 2.3.2. APIs for Decoding There are four basic and easy WMT JDEC APIs for user to decode one JPEG file. int wmt_jdec_buf( vd_handle_t *handle, unsigned char *buf_in, unsigned int bufsize, jdec_frameinfo_t *frameinfo); int wmt_jdec_file( vd_handle_t *handle, WonderMedia? Technologies Inc. November 27, 2009 2 0 char *filename, jdec_frameinfo_t *frameinfo); int wmt_jdec_buf_in_size( vd_handle_t *handle, unsigned char *buf_in, unsigned int bufsize, unsigned int request_w, unsigned int request_h, jpeg_scale_alg scale_alg, jdec_frameinfo_t *frameinfo); int wmt_jdec_file_in_size( vd_handle_t *handle, char *filename, unsigned int request_w, unsigned int request_h, jpeg_scale_alg scale_alg, jdec_frameinfo_t *frameinfo); The common point among these four APIs is the last parameter “jdec_frameinfo_t *frameinfo”, which means a physically continuous frame buffer. Users must know that this frame buffer was allocated automatically inside these APIs, so it must be release by users by calling API wmt_jdec_fb_free() when this frame buffer is useless. int wmt_jdec_fb_free (vd_handle_t *handle, jdec_frameinfo_t *info); All information in this frame buffer are set by driver and returned to user. In general, user should not modify it. Following shows the simplest sample code to decode JPEG file. int main(void) { vd_handle_t handle; jdec_frameinfo_t jfb; /* Initialization */ wmt_init(&handle); /* Start decoding if you have file path */ wmt_jdec_file(&handle, filename, &jfb); /* Or, start decoding if you have a buffer */ /* Display the JPEG */ wmt_jdec_show_image(&handle, &jfb, JPEG_DM_BEST_FIT); /* Free JPEG frame buffer */ wmt_jdec_fb_free(&handle, &jfb); /* Release resource */ wmt_exit(&handle); return 0; } About wmt_jdec_show_image (), we will introduce it in next subsection. WonderMedia? Technologies Inc. November 27, 2009 2 1 In this sample code, it uses all default settings for JPEG decoder. In this case, user does not know what scaling ratio and decoded color are. This picture will be best fit to screen. (The scaling ratio is automatically generated by WMT API according to source dimension and resolution of display device.) You may replace “wmt_jdec_file()” and “wmt_jdec_buf()” in the simplest sample code above by “wmt_jdec_file_in_size()” and “wmt_jdec_buf_in_size()”. The only difference between these two scenarios is in scaling ratio. Scaling ratio of second case is generated according to your input request_w and request_h. Inside these two WMT API in second case, the performance is worse since it does twice frame buffer “copy”, one is by “VDU”, another is “recursive scaling” by “SCALER”. Although the performance is worse, these two APIs are useful for user to create “Thumbnail”. For example, if you want to decode any JPEG to a 256x192 thumbnail, you just need to call wmt_jdec_file_in_size(&handle, filename, 256, 192, JPEG_SA_SMART, &jfb); or wmt_jdec_buf_in_size(&handle, buf, bufsize, 256, 192, JPEG_SA_SMART, &jfb); In theory, these values “request_w” and “request_h” could be arbitrary number. However, 16-alignment is recommended and these two values must be larger than 64 because of hardware limitation. If they are not 16-alignment, the final result may be a little difference with request values. For example, if these two values are 157 and 133, the result generated by hardware may be 144 and 128. 2.3.3. About VDU Scaling Recall the scaling algorithm (“JPEG_SA_SMART, JPEG_SA_QUALITY, or JPEG_SA_FAST”) mentioned in previous subsection. This scaling algorithm only impacts VDU scaling ratio. If application does not call “wmt_jdec_set_scale( )” or call it with “SCALE_BEST_FIT” as input argument before calling “wmt_jdec_file( )”, then the default scaling algorithm in “wmt_jdec_file( )” is “”JPEG_SA_SMART”. int wmt_jdec_set_scale(vd_handle_t *handle, vd_scale_ratio ratio) If application calls “wmt_jdec_set_scale( )” with input argument “SCALE_ORIGINAL, SCALE_HALF, SCALE_QUARTER or SCALE_EIGHTH,” before calling “wmt_jdec_file( )”, then VDU will decode and scale it with this setting. It is not difficult to understand it. Figure 10 – Decode to original dimension WonderMedia? Technologies Inc. November 27, 2009 2 2 Figure 11 – Decode to 1/4 dimension of original dimension Now let us focus on best fit mode case. For best fit mode, how JPEG driver decide exact scaling ratio for hardware for “JPEG_SA_SMART” scaling algorithm? We introduce it in previous subsection. The key point is “Distance” between source dimensions with display resolution. For example, assume there is a JPEG file with dimension 4000x4000. 2000x2000 1000x1000 500x500 2000x2000 1000x1000 500x500 1024x768 640x480 Figure 12 – VDU scaling algorithm If resolution of current display area is 1024x768, the nearest distance to 1024x768 is 1000x1000. Then VDU will decode and scaling to 1/4 automatically in best fit mode. If resolution of current display area is 640x480, the nearest distance to 640x480 is 500x500. Then VDU will decode and scaling to 1/8 automatically in best fit mode. This example shows how “JPEG_SA_SMART” algorithm works inside WMT API “wmt_jdec_file( )”. The behavior of “wmt_jdec_buf( )” is the same with “wmt_jdec_file( )” besides some different input arguments in these two APIs. But how “wmt_jdec_file_in_size( )” works? In fact, there are 3 major steps in this function. wmt_jdec_file_in_size( vd_handle_t *handle, char *filename, unsigned int request_w, unsigned int request_h, jpeg_scale_alg scale_alg, jdec_frameinfo_t *frameinfo) = wmt_jdec_set_scale ( ) + wmt_jdec_file( ) + wmt_jdec_stretch( ) An exact scaling ratio will be calculated according to “scale_alg” and fill it in “wmt_jdec_set_scale( )” “wmt_jdec_file( ) requests VDU to do scaling with this value (1/1, 1/2, 1/4 or 1/8) WonderMedia? Technologies Inc. November 27, 2009 2 3 “wmt_jdec_stretch( )” scales the decoded frame buffer to requested dimension (request_w x request_h) (it was called by recursive scaling mode). We discussed the case above if “scale_alg” is equal to “JPEG_SA_SMART”. Essentially, “JPEG_SA_ SMART” is equal to “JPEG_SA_FAST” or “JPEG_SA_QUALITY” according to “Distance”, which is the same concept above. For example, assume there is a JPEG file with dimension 4000x4000 and requested in “wmt_jdec_file_in_size( )” is 1280x1024. 4000x4000 2000x2000 1000x1000 500x500 Requested dimension VDU Scaling to (JPEG_SA_QUALITY) VDU Scaling to (JPEG_SA_FAST) Figure 13 – Idea of scaling algorithm [Scenario 1] “scale_alg” = “JPEG_SA_FAST”: VDU uses 1/4 scaling ratio, which is smaller and nearest to requested dimension. [Scenario 2] “scale_alg” = “JPEG_SA_QUALITY”: VDU uses 1/2 scaling ratio, which is larger and nearest to requested dimension. [Scenario 3] “scale_alg” = “JPEG_SA_SMART”: VDU uses 1/4 scaling ratio (same as “JPEG_SA_FAST” in this case) because the “Distance” of scenario 1 is nearer to requested dimension than scenario 2. SCLAER System Memory VDU CPU Decoded Frame Buffer Scaled Frame Buffer Source Data Buffer Input Device 4000x4000 2000x2000 768x768 wmt_jdec_file_in_size(handle, filename, 1024, 768, JPEG_SA_QUALITY, fb); WonderMedia? Technologies Inc. November 27, 2009 2 4 Figure 14 – Example of wmt_jdec_file_in_size() From this figure above, the last variable “fb” in “wmt_jdec_file_in_size( )” points to “Scaled Frame Buffer”. Why its dimension is 768x768 instead of 1024x768? Because the shape of original JPEG file is “square”. To keep its shape without torsion, we must decode it to 1024x1024 or 768x768. However, 1024x1024 is out of size than user’s requirement. Therefore, we decode it to 768x768. As for what users see, it depends on the settings in related display APIs. We will introduce it in detail in next subsection. LOOK OUT: the “Decoded Frame buffer” was allocated and released inside “wmt_jdec_file_in_size( )”. User does not know its existence. But the rest memory in MB pool must be larger than 2000x2000 + 768x768 in this example. Otherwise this API will return error because of memory shortage. From WMT API 4.1.1 or later version, we create a new algorithm called “JPEG_SA_HIGH_QUALITY”. This value will forces VDU decodes JPEG to original size without any scaling down. However, there is a risk of memory shortage. In this example, to decode this JPEG file to original size (4000x4000) in ARGB mode, the requested memory is 64 MB. It is huge. Therefore, if memory shortage, “JPEG_SA_HIGH_QUALITY” will be downgrade to “JPEG_SA_QUALITY” automatically inside “wmt_jdec_file_in_size( )”. The behavior of “wmt_jdec_buf_in_size( )” is the same with “wmt_jdec_file_in_size( )” besides some different input arguments in these two APIs. 2.3.4. Real Time and Recursive Scaling We introduced “Real time scaling” and “recursive scaling”. We will discuss it in detail in this subsection. Following figure shows an example of real time scaling. Figure 15 – Flow of real time scaling In this example above, we find that the memory bandwidth is consumed by “VPU”, “GOVW” and “GORV”, respectively. Assume the video format on all frame buffers are ARGB (RGB 32 bits). VPU: VPU consumes at least 457.76 MBytes per second roughly [ (2000x2000x4 (read from decoded frame buffer)

  1. N (scale 2000x2000 to 1024x768 runtime)) x 30 ] (Please keep in mind that hardware needs more memory bandwidth in scaling down than scaling up) GOVW: GOVW read two buffers (both video and graphic) and write one buffer with 1024x768 30 times in one second. Then GOVW consumes 247.5 MBytes per second roughly [ (1024x768x4 (read from graphic plane) + 768x768x4 (read from video plane) +1024x768x4 (write))x30 ] (Here we suppose VPU scales 2000x2000 to 768x768) WonderMedia? Technologies Inc. November 27, 2009 2 5 GOVR: GOVR read one buffer with 1024x768 60 times in one second. Then GOVR consumes 90 MBytes per second roughly [ (1024x768x4 (read from display frame buffer) x60 ] So the roughly total memory bandwidth consumption is 457.76 + 247.5 + 90 + 30xN > 795.26 MB/sec. Now let us use recursive scaling mode in the same example. Following figure shows this case. Figure 16 – Flow of recursive scaling + real time scaling Again, memory bandwidth is consumed by “VPU”, “GOVW” and “GORV”. They are respectively. Assume the video frame on frame buffer is ARGB (RGB 32 bits) VPU: VPU consumes 67.5 MBytes per second roughly [ 768x768x4 (read from scaled frame buffer) x 30 ] GOVW: GOVW read two buffers (both video and graphic) and write one buffer with 1024x768 30 times in one second. Then GOVW consumes 247.5 MBytes per second roughly [ (1024x768x4 (read from graphic plane) + 768x768x4 (read from video plane) +1024x768x4 (write))x30 ] (Here we suppose VPU scales 2000x2000 to 768x768) GOVR: GOVR read one buffer with 1024x768 60 times in one second. Then GOVR consumes 90 MBytes per second roughly [ (1024x768x4 (read from display frame buffer) x60 ] So the roughly total memory bandwidth consumption is 67.5 + 247.5 + 90 = 405 MB/sec. In this example, we find the total memory bandwidth of using recursive scaling mode is about 51% (=405/795.26) of no recursive scaling case. (let N=0). Although memory bandwidth is reduced enormously by using recursive scaling mode, the side-effect is that performance is reduced because we must create a new scaled frame buffer. It is a trade-off and depends on customer’s choice. The key point of recursive scaling in WMT JDEC API is “wmt_jdec_stretch( )”, which was called by “wmt_jdec_file_in_size( )” inside. int wmt_jdec_stretch( vd_handle_t *handle, unsigned int request_w, unsigned int request_h, jpeg_scale_alg scale_alg, jdec_frameinfo_t *jfb_in); The input arguments of this API is same with ones of “wmt_jdec_file_in_size( )”. WonderMedia? Technologies Inc. November 27, 2009 2 6 Since recursive scaling is called “recursive”, it means user may scale it many times as they want. For example, assume someone has a decoded frame buffer with 2000x2000 dimension and he/she wants to scale it down to 300x200. Sample code below shows how to do it.

    Scenario 1: One time recursive> wmt_jdec_stretch(handle, 300, 200, JPEG_SA_QUALITY, &fb);

    Scenario 2: Multi-recursive> wmt_jdec_stretch(handle, 1000, 1000, JPEG_SA_QUALITY, &fb); wmt_jdec_stretch(handle, 500, 400, JPEG_SA_QUALITY, &fb); wmt_jdec_stretch(handle, 300, 200, JPEG_SA_QUALITY, &fb); In these two scenarios, although the last input argument in each API call is same, i.e., “fb”, its essential was modified inside “wmt_jdec_stretch( )” and the original old memory buffer was released in this API, too. Therefore, if you want to keep the original frame buffer, please allocate another frame buffer and copy the data of original frame buffer to new frame buffer. 2.4. Display JPEG There are two APIs to display a decoded JPEG file, which data are stored in a “jdec_frameinfo_t” structure. int wmt_jdec_show_image( vd_handle_t *handle, jdec_frameinfo_t *frameinfo, jpeg_disp_mode mode); int wmt_jdec_show( vd_handle_t *handle, jdec_frameinfo_t *frameinfo, vdo_view_t *view); Basically hardware JPEG has no capability of display. They just call WMT VPP API inside these two APIs. Of course, user may call WMT VPP API directly instead of WMT JDEC API. However, it is better to call WMT JDEC API if your application is JPEG related. Next, we will introduce these two APIs in detail. 2.4.1. Display Mode Enum: jpeg_disp_mode Syntax typedef enum { JPEG_DM_BEST_FIT, JPEG_DM_Original, JPEG_DM_FULL_SCREEN, JPEG_DM_FULL_SCREEN_TORSION, JPEG_DM_MAX } jpeg_disp_mode; /* DM = Display Mode */ Description JPEG_DM_BEST_FIT: If picture is smaller than display area, it will be located on center of screen without scaling up. Otherwise, it will be scaled down to fit display area without torsion (remain aspect ratio). JPEG_DM_Original: Picture will be put at top-left of screen (0,0). If the picture is WonderMedia? Technologies Inc. November 27, 2009 2 7 larger than display area, some part will be invisible. JPEG_DM_FULL_SCREEN: If picture is smaller than display area, it will be scaled up without torsion (remain aspect ratio). If it is larger than display area, it will be scaled down to fit display area without torsion (remain aspect ratio). JPEG_DM_FULL_SCREEN_TORSION: If picture is smaller than display area, it will be scaled up with torsion (not remain aspect ratio). If it is larger than display area, it will be scaled down to fit display area with torsion (not remain aspect ratio).. Note This enumeration is defined in WMT API (wmt-jdec.h). Following figures shows visual result about these four kinds of display mode in two cases. If picture is smaller than display area: In this case, visual result of these four display mode are different each other. Figure 17 – Display mode: if picture is smaller than display area If picture is larger than display area: In this case, visual result of “Best Fit” and “Full Screen” are the same. In “Original” mode, some part, which is out of display area, is cut to keep its original size. Display area Source size JPEG_DM_BEST_FIT Display area JPEG_DM_FULL_SCREEN Display area JPEG_DM_Original Display area JPEG_DM_FULL_SCREEN_TORSION WonderMedia? Technologies Inc. November 27, 2009 2 8 Figure 18 – Display mode: if picture is larger than display area 2.4.2. WMT View Structure These four display modes mentioned above are satisfied for almost applications. However, these are not enough for some applications. To support all kinds of possibilities, we define a structure named “vdo_view_t”. Structure: vdo_view_t Syntax typedef struct { unsigned int resx_src; unsigned int resy_src; unsigned int resx_virtual; unsigned int resy_virtual; unsigned int resx_visual; unsigned int resy_visual; unsigned int posx; unsigned int posy; unsigned int offsetx; unsigned int offsety; } vdo_view_t;; This structure describes all settings of one frame buffer used for JPEG driver only. This structure should be thought as a “subclass” of vdo_framebuf_t class. resx_src: this value is similar with fb_w in vdo_framebuf_t. It means line size. resy_src: this value is similar with fb_h in vdo_framebuf_t. resx_virtual: this value is source width which user wants to be shown Display area Source size JPEG_DM_BEST_FIT Display area JPEG_DM_FULL_SCREEN Display area JPEG_DM_Original Display area JPEG_DM_FULL_SCREEN_TORSION WonderMedia? Technologies Inc. November 27, 2009 2 9 resy_virtual: this value is source height which user wants to be shown resx_visual: this value is destination width on display device resy_visual: this value is destination height on display device offsetx: means the start x-coordinate of source image which user wants to be shown offsety: means the start y-coordinate of source image which user wants to be shown posx: means the start x-coordinate of destination on display device posy: means the start y-coordinate of destination on display device Note This structure is defined in and shared with WMT API (com-video.h). Developers may refer to figure below to understand in detail about this structure. Figure 19 – vdo_view_t introduction In almost applications, the values of (offsetx, offsety) are always (0, 0). Moreover, the values of (resx_virtual, resy_virtual) and (resx_src, resy_src) are always the same with (img_w, img_h) and (fb_w, fb_h) in “vdo_framebuf_t” or “jdec_frameinfo_t”. As for (posx, posy) and (resx_visual, resy_visual), they depends on relationship between source dimension with resolution of display device. For example, assume the resolution of display device is 1024x768, source image is 400x300. To scale up full image two times and show it at center on display device, you must set /* Source */ resx_src = 448; /* 64 alignment of 400 */ resy_src = 304; /* 16 alignment of 300 */ offsetx = 0; offsety = 0; resx_virtual = 400; resy_virtual = 300; source resy_src Display Area resx_src resx_visual resy_visual posx posy offsetx offsety resx_virtual resy_virtual WonderMedia? Technologies Inc. November 27, 2009 3 0 /* Destination */ resx_visual = 800; /* 400x2 */ resy_visual = 600; /* 300x2 */ posx = 112; /* (1024-800)/2 */ posy = 84; /* (768-600)/2 */ 2.5. Advance JDEC Features 2.5.1. Allocate Physically Continuous Memory In WonderMedia? API architecture, all physically continuous memory related with video frame buffer must be handled by MB (Memory Block) driver. For application, it may call WMT MB API to allocate/free this kind of memory. However, we do not recommend user to do that because there are some detail must be deal with, i.e., alignment issue. Therefore, we provide following APIs in WMT JDEC API for applications. int wmt_jdec_fb_alloc ( vd_handle_t *handle, unsigned int pic_w, unsigned int pic_h, vdo_color_fmt src_color, vdo_color_fmt decoded_color_in, jdec_input_t *arg_in); int wmt_jdec_fb_free (vd_handle_t *handle, jdec_frameinfo_t *info); int wmt_jdec_alloc ( vd_handle_t *handle, unsigned int buf_w, unsigned int buf_h, vdo_color_fmt color_fmt, jdec_frameinfo_t *jfb); int wmt_jdec_free (vd_handle_t *handle, jdec_frameinfo_t *jfb); Basically these two free APIs are identical essentially since all parameters in them are the same. The major difference between two allocate APIs is in returned value, “jdec_input_t” and “jdec_frameinfo_t”, respectively. The structure “jdec_input_t” is used inside some WMT JDEC API. Therefore, developer do not need to understand it too mush. They just need to know when wmt_jdec_fb_alloc( ) returned, following members in “jdec_input_t” would be filled automatically. dst_y_addr_user: User space address of Y or RGB plane dst_c_addr_user: User space address of C or RGB plane dst_y_addr: Physical address of Y or RGB plane dst_c_addr: Physical address of C plane (Null for RGB) dst_y_size: Buffer size for Y or RGB plane dst_c_size: Buffer size for C plane (0 for RGB) The structure “jdec_frameinfo_t” is very popular among WMT JDEC API. In application developer’s point of view, calling “wmt_jdec_alloc ()” to allocate physically continuous memory is recommended if they need to do that. (LOOK OUT: “wmt_jdec_alloc ()” is supported in WMT API version “4.0.3” or later version) LOOK OUT: Once application gets a physically continuous frame buffer, only user-space address could be accessed by application, i.e., “dst_y_addr_user/dst_c_addr_user” in “jdec_input_t” or “y_addr_user/c_addr_user” in “jdec_frameinfo_t” since all application under Linux running in user space. Besides, this memory comes from WonderMedia? Technologies Inc. November 27, 2009 3 1 “Allocated Memory by MB” and it has no relationship with “Memory Pool For Linux kernel & application” in Figure 7. In other word, user cannot find memory shortage if he/she type “free” or “cat /proc/meminfo” Linux command even user had allocated the video frame buffer. To watch MB memory status, please type “cat /proc/mbinfo”. More detail about MB, please refer to another chapter “Memory Block (WMT MB)”. 2.5.2. MSD (Multi-Segment Decoding) We discussed memory usage in previous subsection. (Please refer to “Figure 8 Data Path of hardware JPEG decoder”) So we know to decode a JPEG file, we need at least two memory buffers, “Source Data Buffer” and “Destination Video FB”, respectively. Basically, the “Source Data Buffer” size is equal to file size of a compressed JPEG file. Although it was compressed, its size is still very huge if its quality is good and resolution is high. If we still request “Source Data Buffer” size must be equal to file size, sometimes JPEG decoding is fail if rest system memory is not enough. To solve this problem, we provide a method named MSD. So-called MSD is that we just allocate a small buffer named “Segment Buffer” to store part of compress JPEG file. For example, if a compress JPEG file is 8.5 MB and segment buffer is 1 MB. Then we must do 9 loops to finish the JPEG decoding work. In each loop JPEG decoder only decodes 1 MB except the last loop (only 0.5 MB). MSD provides a good way to decode JPEG when system memory is not enough. However, the performance becomes a little worse if MSD is enabled. As for how large is better for segment buffer? Since the larger the size of segment buffer is, the better the performance is. We recommend it is 1 MB or larger. To enable/disable MSD, please use following API. Syntax int wmt_jdec_msd_enable( vd_handle_t *handle, unsigned int fileszie, unsigned int segment_size, unsigned int msd_enable) Parameters handle [IN]: means the handle of JPEG decoder fileszie [IN]: means the file size of JPEG file. Or set “0” to ignore it. segment_size [IN]: means size of segment buffer, 1 MB or larger is recommended. msd_enable [IN]: means the MSD flag (enable:1 or disable: 0). Description The API is used to enable/disable MSD for WMT JDEC API. Return value Return 0 if success; otherwise return an error code (negative value) Note Two JDEC APIs, “wmt_jdec_file()” and “wmt_jdec_file_in_size()”, are affected by this API. Sample codes below shows how to enable/disable MSD. int main(void) { vd_handle_t handle; jdec_frameinfo_t jfb; /* Initialization */ wmt_init(&handle); /* Enable MSD */ wmt_jdec_msd_enable(&handle, 0, 1024*1024, 1); /* Start decoding if you have file path */ WonderMedia? Technologies Inc. November 27, 2009 3 2 wmt_jdec_file(&handle, filename, &jfb); /* Or, start decoding if you have a buffer */ /* Display the JPEG */ wmt_jdec_show_image(&handle, &jfb, JPEG_DM_BEST_FIT); /* Free JPEG frame buffer */ wmt_jdec_fb_free(&handle, &jfb); /* Release resource */ wmt_exit(&handle); return 0; } Besides “wmt_jdec_msd_enable ()”, there is another API, “wmt_jdec_segment_write()”, for MSD if user does not want use “wmt_jdec_file()” and “wmt_jdec_file_in_size()” to decode JPEG. Syntax int wmt_jdec_segment_write( vd_handle_t *handle, unsigned char *buf_in, unsigned int bufsize, unsigned int filesize, unsigned int seg_no) Parameters handle [IN]: means the handle of JPEG decoder buf_in [IN]: means the address pointed to segment buffer bufsize [IN]: means size of segment buffer, 1 MB or larger is recommended filesize [IN]: means the file size of JPEG file seg_no [IN]: means the segment number (start from 1, not 0) Description The API is used for MSD only Return value Return 0 if success; otherwise return an error code (negative value) Note Following sample codes shows this new way for MSD. int main(void) { vd_handle_t handle; jdec_frameinfo_t jfb; /* Initialization */ wmt_init(&handle); /* Enable MSD */ seg_size = 1024 * 1024; wmt_jdec_msd_enable(&handle, filesize, seg_size, 1); /* Start decoding if you have file path */ segment = 1; remain_byte = filesize; WonderMedia? Technologies Inc. November 27, 2009 3 3 while(remain_byte) { if( remain_byte <= seg_size) { read_size = remain_byte; remain_byte = 0; } else { read_size = seg_size; remain_byte -= seg_size; } if( read_size != fread(buf, 1, read_size, fp) ){ /* Do error handling here */ } ret = wmt_jdec_segment_write(handle, buf, read_size, filesize, segment); if( ret != 0 ){ /* Do error handling here */ } segment++; } /* Wait for decdoing finish after all data are feed */ ret = wmt_jdec_finish(&handle, &jfb); /* Display the JPEG */ wmt_jdec_show_image(&handle, &jfb, JPEG_DM_BEST_FIT); /* Free JPEG frame buffer */ wmt_jdec_fb_free(&handle, &jfb); /* Release resource */ wmt_exit(&handle); return 0; } LOOK OUT: If developers use this way for MSD, the second parameter, “filesize”, must be filled correctly. Otherwise, the decoding will fail in driver side. 2.5.3. More Detail about JDEC We had introduced two ways to decode JPEG in previous subsections. If user is expert in WMT JDEC API, he/she may use the skill mentioned in this subsection to implement his/her application. Get JPEG header information: If user wants to know information about JPEG properties, it is better to call WonderMedia? API to decode header at first. int wmt_jdec_header ( unsigned char *buf_in, unsigned int buf_size, jdec_hdr_info_t *hdr); If user knows all information about JPEG file, e.g., image width (sof_w), image height (sof_h), source color format (4:2:0/4:2:2/4:4:4) and file size, he or she will not need to decode header again. It is better to call following one of two API set for hardware decoder. int wmt_jdec_set_header(jdec_hdr_info_t *hdr); or WonderMedia? Technologies Inc. November 27, 2009 3 4 int wmt_jdec_set_src_color(vdo_color_fmt src_color); int wmt_jdec_set_sof( unsigned int sof_w, unsigned int sof_h, unsigned int filesize); Prepare all necessary information: Before starting to hardware decoder, user must set necessary setting for decoder, especially scaling mode and color format on decode frame buffer. It is better to set decoding information at least once. Otherwise, WonderMedia? API will use default values, which may be what user wants. int wmt_jdec_set_decode_info(jdec_decode_info_t *di_in); or int wmt_jdec_set_scale(vd_scale_ratio ratio); int wmt_jdec_set_dc_format(vdo_color_fmt decoded_color); Allocate memory as decoded frame buffer: Because of hardware decoder limitation, the decoded frame buffer should be physical continuous. Namely, user can not use “malloc( )” to allocate decoded frame buffer. To allocate physical address memory in user space on Linux, we implement MB (Memory Block) APIs to achieve it. Fortunately, ND application developers do not need to know the detail, we encapsulate them into two APIs. int wmt_jdec_fb_alloc ( unsigned int pic_w, unsigned int pic_h, vdo_color_fmt src_color, vdo_color_fmt decoded_color_in, jdec_input_t *arg_in); int wmt_jdec_fb_free (jdec_frameinfo_t *info); Once the frame buffer is useless, please do not forget to release it by calling wmt_jdec_fb_free( ). Enable hardware decoding: After setting header information and decoder information, user must call a “fire” wmt_jdec_attr_ready( ) function and wmt_jdec_proc ( ) to enable hardware decoding. int wmt_jdec_attr_ready(void); int wmt_jdec_proc(jdec_input_t *arg_in); Wait hardware decoding finished: Once user enable hardware decoding, he or she must call following API to wait for decoding finish. int wmt_jdec_finish(jdec_frameinfo_t *frameinfo); One thing user must know is that almost time spent in this process is in wmt_jdec_finish( ). In other word, the decoding job is finished only when wmt_jdec_finish( ) is returned instead wmt_jdec_proc ()is returned. If performance is very critical, developers may implement their application as multi-thread, but they must consider the overhead of context switch. wmt_jdec_finish(frameinfo); /* thread is blocking in driver until decoding finished */; WonderMedia? Technologies Inc. November 27, 2009 3 5 After wmt_jdec_finish() returned, user will get information about decoded frame buffer, which is stored in “frameinfo”. Then user may display this frame according to “frameinfo”. Display decoded JPEG frame buffer: The API for user to display a frame buffer is int wmt_jdec_show_image( vd_handle_t *handle, jdec_frameinfo_t *frameinfo, jpeg_disp_mode mode); int wmt_jdec_show( vd_handle_t *handle, jdec_frameinfo_t *frameinfo, vdo_view_t *view); Although the name of these two APIs with prefix “jdec”, it is also workable for any raw data as long as its meets “jdec_frameinfo_t” structure. For example, user may copy a BMP file to a “jdec_frameinfo_t” structure and then use these two APIs to display BMP file. 2.5.4. JPEG Rotation Rotating JPEG file is necessary for almost multimedia products. WonderMedai? SoC also supports this feature. However, “Rotation” is a feature of “GE” instead of one of video decoders. The basic concept about JPEG rotation could be divided into stages below. Decide the decoded dimension according to rotation angle and resolution of display device Decode JPEG to desired dimension computed above Allocate new frame buffer as “GE” destination buffer to save rotated image data Use “GE” to rotate image Free original JPEG frame buffer Display rotated image Free frame buffer used to save rotated image data Sample codes below teach user how to rotate data by “GE”. static int wmt_rotate( vdo_framebuf_t *dst_fb, vdo_framebuf_t *src_fb, unsigned int rot_degree) { unsigned int args[6]; int fd; int ret; unsigned int cmd; if( (src_fb->col_fmt != VDO_COL_FMT_ARGB) || (dst_fb->col_fmt != VDO_COL_FMT_ARGB)) { DBG_ERR("Only supoprt ARGB now!\n"); return -1; } WonderMedia? Technologies Inc. November 27, 2009 3 6 cmd = _IOW(0x69, 3, void *); /* GEIO_ROTATE */ args[0] = src_fb->y_addr; /* phy_src */ args[1] = dst_fb->y_addr; /* phy_dst */ args[2] = src_fb->fb_w; /* width */ args[3] = src_fb->fb_h; /* height */ args[4] = src_fb->bpp; /* bpp */ args[5] = rot_degree; /* arc */ fd = open("/dev/fb0", O_RDWR); ret = ioctl(fd, cmd, args); close(fd); return ret; } /* End of wmt_rotate() */ Currently we do not define any WMT API about rotation. Maybe we will package the sample codes above as standard WMT API in later version. 2.5.5. PD (Partial Decoding) There is always a tradeoff between image quality with performance. To get better quality, system needs more memory to decode it without scaling down. In this case, the decoding performance will become worse. However, sometimes people think image quality is most important even worse performance. In this condition, we cannot let “VDU” to scaling down JPEG during decoding. In other words, we must let “VDU” to decode JPEG to original size. This creates a new problem. Is the physical continuous memory always enough for hardware to decode an original picture size? For example, if there is a JPEG with 8192x8192 dimensions, to decode it into ARGB color format, we need 8192*8192*4 = 268,435,456 = 256 MB. It is impossible for embedded system. To solve this problem, we provide a new concept called “Partial Decoding”. As we introduce above, to decode one JPEG frame/file by WonderMedia? hardware, user needs two memory buffers, “Source Data Buffer” and “Destination Video FB”. We tried to use MSD technology to save size of “Source Data Buffer”. Now we will use “PD (Partial Decoding)” to save size of “Destination Video FB”. The basic concept of PD is that we just decode some part area which we want one time. There are two usual user scenarios about it. Just to show some part of JPEG on screen For example, suppose the picture size is 4096x4096 and resolution of display device is 1024x768. If user wants to show picture with original size on screen, i.e., only 1024x768 of 4096x4096 is displayed, we may just decode 1024x768 instead of full picture. The memory size of “Destination Video FB” will be reduced from 64 to 3 MB in ARGB mode. Original Size: 4096x4096x4 = 64 MB PD Enable: 1024x768x4 = 3 MB Usually user does want to see not only left-top of picture but also other area. Therefore, this technology of this case may be used in “scroll” function of picture if applications have scroll bars at right and bottom sides of screen. Another case is “zoom” function to adopt this technology. WonderMedia? Technologies Inc. November 27, 2009 3 7 Figure 20 – First scenario of PD Multiple decoding JPEG strip by strip Figure 21 – Second scenario of PD Display area Source size VDU Destination Buffer PD enabled Final Destination Video FB Recursive Scaling Final display WonderMedia? Technologies Inc. November 27, 2009 3 8 The second scenario is used for user who wants best quality but lower performance is acceptable. Assume source size of JPEG picture is 4096x4096 and resolution of display device is 1024x768. If we have only 3 MB physically continuous memory buffer as “VDU Destination Buffer”, then the maximum height of strip is (3x1024x1024)/(4096x4) = 192 pixels. So, we may decode this file 24 times (4096/192). The first decoding is from 0 ~ 191 in height, the second is 192 ~ 383…. Besides “VDU Destination Buffer”, we also need a buffer “Final Destination Video FB” (with dimension 1024x768). The basic assumption of this way is that the quality of image with twice scaling by VDU and VPU is worse than one with one scaling by VPU directly. It is true, if we let “VDU” to scaling image down, then image quality will lost. Of course, “VPU” use the lost quality to scaling again, the quality must be worse. Although the quality is better, the performance must be worse. To learn to use PD, “ut_jdec” is a good sample. Next, we will introduce how to use PD function. Following structure and API are used for PD only. Structure: jdec_pd Syntax typedef struct { unsigned int enable; unsigned int x; unsigned int y; unsigned int w; unsigned int h; } jdec_pd; /* PD: Partial Decode */ Description enable: It used to enable hardware PD (Partial Decoding) or not x: The value means X of start decode (16xN alignment) (Unit: Pixel) y: The value means Y of start decode (16xN alignment) (Unit: Pixel) w: The value means Width of start decode (16xN alignment) (Unit: Pixel) h: The value means Width of start decode (16xN alignment) (Unit: Pixel) Note This structure is defined in WonderMedia? API (com-jdec.h). int wmt_jdec_set_pd(vd_handle_t *handle, jdec_pd *pd_in); /* default: Diable PD function */ Sample code below shows use PD to decode 1024x768 area of a large picture. int main(void) { vd_handle_t handle; jdec_frameinfo_t jfb; jdec_pd pd; /* Initialization */ wmt_init(&handle); /* Enable PD */ pd.enable = 1; pd.x = pd.y = 0; pd.w = 1024; pd.h = 768; wmt_jdec_set_pd(&jdec_handle, &pd); /* Start decoding if you have file path */ wmt_jdec_file(&handle, filename, &jfb); /* Or, start decoding if you have a buffer */ WonderMedia? Technologies Inc. November 27, 2009 3 9 /* Display the JPEG */ wmt_jdec_show_image(&handle, &jfb, JPEG_DM_BEST_FIT); /* Free JPEG frame buffer */ wmt_jdec_fb_free(&handle, &jfb); /* Release resource */ wmt_exit(&handle); return 0; } 2.6. Motion JPEG (MJPG) Inside WonderMedia? hardware design, they are the same to decode a JEPG picture with Motion JPEG movie. They both need “VDU” to decode a JPEG frame. In other words, developers can not decode a JEPG file and MJPEG at the same time since there is only one hardware video decoder. Figure below shows the software architecture for JPEG and MJPEG. WonderMedia? API JPEG Application Media Player (MPEG Application) Figure 22 – Software architecture for JPEG and MJPEG In fact, the major difference between JPEG picture and motion JPEG movie, it is in application’s behaviors. It is not difficult to decode a JPEG file as long as following this document mentioned above. However, to play a MJPEG movie, developer must have a “Media Player” to do more things besides decoding JPEG frame. For example, “to demux media file header”, “to separate audio/video bitstreams”, “AV synchronization” and so on. They are all dependent on the behavior of media player which developers use. One thing must be kept in mind is that it is the same for WMT JDEC to decode JPEG or MJPEG. In WMT JDEC’s point of view, it just decodes a JPEG frame. (MJPG is a set with many JPEG frames.) WMT JDEC does nothing besides decoding JPEG frame. Other works, e.g. “AV synchronization” or “Scaling to fit screen”, should be done by media player. It is nonsense to scaling JPEG frame when playing a movie. If developers want to scaling video data of a movie, “Real time scaling” is the best choice instead of “VDU” scaling. To demonstrate MJPEG, WonderMedia? formal BSP adopts “MPlayer” as the media player. (Please refer to “1.5 Open Source Licenses” in detail about “MPlayer”) Of course, developers may use any media player as they want, but there is a little porting effort on it. Figure below shows software architecture of MPlayer for MJPEG. WonderMedia? Technologies Inc. November 27, 2009 4 0 WonderMedia? API JPEG/ MJPEG OSS User Space Kernel Space JPEG Driver (VDU) Audio Driver (Output) FFMPEG (Video) FFMPEG (Audio) FFMPEG Adapter MPlayer MPlayer Figure 23 – Software architecture of MPlayer for MJPEG We will not discuss too much about audio path. We just focus on JPEG path only here. If your media player supports FFMPEG, it is an easy job to port your media player to WonderMedia? platform with hardware decoding feature because WonderMedia? create an interface called “FFMPEG adapter” to connect WMT API with player. If not, developers must create a new interface like “FFMPEG adapter” to connect WMT API with their player. It is case by case, we will not discuss it in detail here. WonderMedia? Technologies Inc. November 27, 2009 4 1


  

トップ   編集 凍結 差分 バックアップ 添付 複製 名前変更 リロード   新規 一覧 単語検索 最終更新   ヘルプ   最終更新のRSS
Last-modified: 2012-09-12 (水) 10:01:54 (2281d)